How to install fluentD plugins with docker-compose / dockerfile combination [closed]

I am trying to install a docker plugin to fluentD in my local docker setup.

My problem is that the plugin installs perfectly but then fluentD will complain on startup:

Step x/x : RUN gem install fluent-plugin-datadog ---> Running in 63c0bca8615bSuccessfully installed connection_pool-2.4.1Successfully installed net-http-persistent-4.0.4Successfully installed fluent-plugin-datadog-0.14.43 gems installed

[error]: config error file="/etc/fluent/fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'datadog'. Run 'gem search -rd fluent-plugin' to find plugins"

This happens with every plugin. I found several online resources that mention this issue.But there is no fix available.I suppose this might be due to some rights issue. Though the file rights of the plugin folder seem okay when I go into the container.

How do I install a plugin in fluentD with docker?

In my docker-compose.yml I add the following for fluentD:

  fluent:    build: ./fluentd    container_name: fluentd    ports:      - 24224:24224    volumes:      - ./fluentd:/fluentd/etc

This way it builds it from the local ./fluentd folder.Then in that folder I have the following Dockerfile:

FROM fluentd:v1.17.1-debian-1.0USER rootCOPY ZscalerRootCA.pem /usr/local/lib/ruby/3.2.0/rubygems/ssl_certs/rubygems.orgRUN gem install fluent-plugin-datadogUSER fluentENTRYPOINT ["tini",  "--", "/bin/entrypoint.sh"]CMD ["fluentd"]

As an example I am installing datadog plugin here. But the issue arises with every plugin I want to install.Finally the fluent.conf will have something like this:

<match *.**>  @type datadog  # rest omitted</match>