Chatbots like ChatGPT demonstrate convincing moral rhetoric and can even outperform humans in perceived morality. However, their lack of understanding and embodied experiences raises concerns about their ability to provide accurate moral guidance. Transparency, context consideration, and ethical implications should be addressed by software companies, data professionals, and governments.