[OpenSIPS-Users] Getting call analytics to Elasticsearch

[Gerwin van de Steeg]

The usual use case of Kafka (or Redis/AMQP) is to sit as the middle man to
buffer and queue the material before it goes into logstash and then elastic.

ie.  Kafka -> Logstash -> Elastic nodes

If you are only pushing the data to Elastic then pushing it via the
logstash HTTP input or a load balanced logstash should be suitable to give
you some form of buffering and queueing of messages.
Or you can skip the intermediate step here and go directly to the event
queue based approach start using Kafka/AMQP or even a simple Redis queue,
where you would then publish your message to these queues and logstash just
pulls them out to populate elastic nodes at the rate it can handle.

Cheers,


Gerwin




On Thu, 26 Nov 2020 at 07:04, Calvin Ellison <calvin.ellison at voxox.com>
wrote:

> Is there any best practice or standard method of getting data from
> OpenSIPS to Elasticsearch?
>
> The scenario is an internal call filter that will block or permit a call.
> e.g. 503 route advance or 603 stop route. Checks are performed against
> pre-populated caches and database queries, and we want the results of those
> checks to ultimately reach Elasticsearch.
>
> I'd prefer to avoid file-based things like syslog + logstash. The
> "OpenSIPS and BigData" presentation from 2016 suggested using the
> Elasticsearch REST API directly with async(rest_post()) or RabbitMQ. Our
> experience with Elasticsearch indicates it will not keep up with our call
> volume without a message queue, and now I'm seeing talking about Kafka
> being the new standard for this.
>
> Is this a solved problem?
>
>
> Regards,
>
> *Calvin Ellison*
> Senior Voice Operations Engineer
> calvin.ellison at voxox.com
> +1 (213) 285-0555
>
> -----------------------------------------------
> *voxox.com <http://www.voxox.com/> *
> 5825 Oberlin Drive, Suite 5
> San Diego, CA 92121
> [image: Voxox]
> _______________________________________________
> Users mailing list
> Users at lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensips.org/pipermail/users/attachments/20201126/4fe955bc/attachment.html>