For a piece of software infrastructure, nothing is more important than good applications to showcase it’s features and demonstrate the possibilities it enables. Fortunately there is a very large body of existing applications because PipeWire implemented the PulseAudio and Jack APIs. This article highlights some of the great efforts undertaken in the community around creating interesting applications for PipeWire in an interview with the maintainers of two of the most popular applications built with PipeWire. They are Wellington Wallace, maintainer of EasyEffects for PipeWire and Tom Wagner, maintainer of the Helvum patchbay application for PipeWire.

Christian Schaller: First of all thank you both for taking the time to talk with me. I am really excited about your projects and I know a lot of people in the PipeWire community are too. So let’s start with introductions, tell us a little about yourself and how you got involved with open source development.

<figure class="wp-block-image"><figcaption>Screenshot the Helvum patchbay in action </figcaption></figure>

Tom Wagner: I am a 7th semester Computer Science student at the Technical University Berlin, currently working on my Bachelors degree. My first real contact with Linux was in my school, where the computers ran Linux Mint.

I started using Linux on my personal devices a few years later, because at that point I already had some experience with it and because I became increasingly annoyed by Windows.

After some smaller contributions to different open source projects, looking for a project to work on, I found PipeWire and started working on Helvum and PipeWires Rust bindings.

Wellington Wallace: I am a physicist currently working as a professor in a Brazilian federal education center called Centro Federal de Educação Tecnológica Celso Suckow da Fonseca. My first contact with Linux happened in  the computational physics lectures I attended at the beginning of my studies. From that moment on Linux  became the only operating system installed on my computer. But it was only some years later that programming became part of my day to day life.

The focus of my PhD was experimental physics. In order to understand the measurements I did about magnetic nanoparticles I started to do Monte Carlo simulations. First in Python and after some time in C++. But, as it was the kind of simulation that took many days to finish, I eventually started to use OpenCL.

My first open source project was a tool to make graphs in my experimental physics classes. But the one where I really felt I was contributing with something unique to the open source community was PulseEffects. This later became EasyEffects when I switched to PipeWire. The project started one day when Pulseaudio suddenly stopped playing audio in one of the channels. I could have tried to fix the problem but the truth was that I wanted more than just an equalizer. That was how PulseEffects was born. Although I expected it to be useful for some people I did not think this project would get so big.

Christian Schaller: Could both of you give us a short introduction to your project and what it does, for those who might be unfamiliar?

Wellington Wallace: EasyEffects(formerly known as PulseEffects) is a tool that allows the user to add global audio effects for applications. Among the possibilities are dynamic range compression, equalization, automatic volume control, noise reduction, bass enhancement, etc. Besides applications output, users can add effects for microphones too.

Tom Wagner: PulseAudio is mostly designed to have applications receive audio input from one single input device and/or send audio to a single output device, which makes it difficult to adapt to more complicated use-cases.

Contrary to PulseAudio, PipeWire runs a full media graph. This means that audio and other kinds of media can be freely routed between any number of devices and even to other applications at the same time.

For example, you could route the audio output of your music streaming software to your speakers as well as your voice-chat software to listen with others, while routing the audio and video of a game to a recording software to make a recording without the music in it.

Helvum is a GUI to do exactly this: create and delete connections between devices and/or programs to route data through the system in exactly the way you want it.

Christian Schaller: Wellington, so EasyEffects started out as an application targeting PulseAudio, but you switched it over to PipeWire relatively early on. What made you decide to jump on and target PipeWire with EasyEffects?

<figure class="wp-block-image"><figcaption>Screenshot of EasyEffects in action</figcaption></figure>

Wellington Wallace: Actually, between the first time users asked for PipeWire to be supported and my move to do that, took about 2 years. It may seem, for some people, that it was early. But almost all that we needed was already available.

Besides the usual “low latency” argument people give when talking about PipeWire, there were other  reasons that were important from an application implementation point of view. Pulseaudio does not offer a way to naturally do the kind of processing we do. There is no support for third party plugins in Pulseaudio that could be used to write filters. What is done in PulseEffects, in a way, is a hack. I had to “take the audio away from the server” so I could process it in GStreamer. After everything was done I had to send the result to Pulseaudio. It may seem simple but that approach had lots of problems with clock drift in the beginning. Audio got out of synchronization so fast that I almost gave up on the project. It took lots of attempts with different GStreamer configurations until I found a way to make the clock drift negligeable.

In PipeWire the whole process feels natural and the filter API is super simple. Way easier than writing GStreamer plugins. Handling the pipeline is a lot easier too. I like GStreamer but sometimes removing or inserting plugins from/to its pipeline on the fly can be a nightmare….

Another thing we get for free on PipeWire is realtime priorities in the filter thread. On Gstreamer I had to handle that myself. That was mostly fine. But, since Flatpak did not have RealtimeKit support, people using PulseEffects through Flatpak could not have realtime priorities in our audio thread.

Another reason that indirectly contributed to the early adoption was the release of gtk4(gtkmm4). The combination of  Pulseaudio and GStreamer did things in a very different way when compared to PipeWire. Moving from  gtkmm3 to gtkmm4, while keeping Pulseaudio and GStreamer, would make me work twice in some places once the move to Pipewire was made. If I had to change the graphical interface anyway it is better to do it once using the new audio server.

Christian Schaller: Tom, so Helvum is a brand new application written explicitly for PipeWire. What was your motivation for starting this project?

Tom Wagner: I used to play World of Warcraft and wanted to record my guilds raids with separated game and voice chat audio tracks so I could mute the voice chat for some clips and keep them for others.

This was fairly complicated and painful with PulseAudio, where I had to do some complicated setup of creating multiple virtual loopback sinks to first record these audio streams separately and later combine them to send them to my headphones.

I was looking for some open source project to contribute to, and became interested in PipeWire, where this could be achieved much easier by rerouting the media graph.

Back then, there was no native PipeWire patchbay, so you had to rely on JACK tools.

While those could be used to route audio and MIDI through a compatibility layer, JACK and therefore any JACK tools can’t handle video, so as my contribution to the open source community I started work on PipeWire Rust bindings and, using those, came Helvum.

Christian: Wellington and Tom, what has your experience been working with Wim Taymans and the PipeWire community so far?

Wellington Wallace: I consider it a good experience. Whenever I need help to get things done in PipeWire Wim Taymans responds reasonably fast. Especially when considering the amount of work he has on his hands now.

Tom Wagner: It’s been great so far. The IRC channel (#pipewire on OFTC), including Wim, is always able to answer any questions that come up quickly, and we already had many great contributions to both Helvum as well as the Rust bindings.

Christian Schaller: Wellington, so what are your plans for EasyEffects going forward? What do you see as the next steps for EasyEffects?

Wellington Wallace: At this moment I am in the process of using gtk4 API directly. GTKMM has served us well but now that I want to use LibAdwaita it will be more straightforward to talk to gtk4 directly. It is not the first time I do something like that. I started this project in Python. Its gtk wrapper is good but eventually it got in the way. And now the same thing is happening with gtkmm.

Besides that there are 2 more things that are on the horizon but for which I do not have a solution yet. The first is adding support for multiple instances of a given filter. For example some people would like to have more than one equalizer instance in the pipeline. At this moment this is not possible. Another rough edge in EasyEffects is the lack of upmixing from stereo to surround devices. But I still think that some of the work required for that would also have to be done on PipeWire.

Christian Schaller: And Tom, where do you plan on taking Helvum from this point forward?

Tom Wagner: At the moment, to configure things such as volume, default audio devices etc. via GUI, the best way is still via PulseAudio applications such as pavucontrol.

I’d like for Helvum to become a full PipeWire management interface, able to handle all these use cases natively.

There are also many quality-of-life improvements such as zooming and better scrolling that are still missing.

Christian Schaller: Tom and Wellington, what do you like the most about PipeWire and do you have any wishes or feature improvements you would like to see from PipeWire?

Tom Wagner: The best thing about PipeWire for me is definitely its flexibility and versatility.

It allows routing data through the system any way you want, even between PulseAudio and JACK applications, without any trouble. It lets power users run their audio with minimal latency, and still remains a drop-in pulseaudio replacement even for casual users.

From my perspective, PipeWire is already in really great shape, and while there can still be some issues, especially with less common setups, it keeps getting better with each release.

Wellington Wallace: Some of my thoughts on that are scattered in my previous comments. It has a simple API that fits our needs and the latency is low enough. Another thing I consider important is a short release cycle. Pulseaudio takes too long to make new releases. I understand that it is an old project by now but I saw in the past some annoying bugs being fixed in Pulseaudio’s development branch that took forever to get to the users. I hope that PipeWire keeps the current approach where releases are done more frequently.

About the improvements, it depends on the point of view. Thinking about it as a user I think PipeWire still has some rough edges when it comes to on-the-fly device connection/disconnection. Like headphones, for example. It still resets volumes to 100% or mutes them in cases that Pulseaudio did not do. It is not often that this happens. But it still happens.

From the EasyEffects point of view I think that the process available to link filters needs some love. Right now if a filter has to link to a surround, or any other device that has a different number of channels, the whole upmixing/downmixing process becomes the audio clients (EasyEffects) responsibility. As I do not have the knowledge for that, the current situation is that there is no upmixing or downmixing being done in these cases. We did a few attempts with the loopback module but it does not fit very well with how we do things. It also generates additional streams that will only confuse the user and increase the chances our audio routing is broken by something that decides to move the loopback streams somewhere they should not be. In an ideal world I would like to have a way to just tell PipeWire “link this filter’s channels to this input/output device and do upmixing/downmixing if necessary”. 

Christian Schaller: So if anyone wants to contribute to either Helvum or EasyEffects what is the best way to get started and get involved?

Wellington Wallace: Up to now 65 people have contributed in some way to EasyEffects(PulseEffects). In all these cases either a pull request was done at https://github.com/wwmm/easyeffects or an issue was opened to discuss things first. In many cases people contributed only with an idea. If it is something that can be done and is compatible with how EasyEffects works, I or someone else usually implements it when time is available. 

Tom Wagner: Similar to EasyEffects, reporting any problems or pet peeves to the issue tracker already goes a long way towards improving the application.

From there, anyone can start work on a Merge Request to fix the issue. Any questions and requests for help are also welcome there, and I’ll do my best to answer them.

Christian Schaller: Are there any projects you would like to see started with PiperWire that you feel are clearly outside the scope of your application?

Tom Wagner: Anything that involves processing the actual data that is being sent is definitely out of scope. For example, people have already asked for support for hosting plugins such as VST and OFX plugins. These would be better handled by a separate app, as the focus for Helvum is on controlling the PipeWire server, not the data being routed through it.

Wellington Wallace: I think that what Helvum is trying to achieve is exactly the kind of thing I have always considered out of EasyEffects scope. I want to focus on providing as many useful effects as possible in a way that is flexible enough for advanced users without making new users too scared. Managing devices, like Pavucontrol does for Pulseaudio and to some extent to PipeWire, is something that another application should do.

Christian Schaller: Tom, from what I know, you are a Fedora Linux user. What has made it your distribution of choice?

Tom Wagner: Yes, after a while of using Arch, I wanted something that works well out of the box, and after trying a number of different distros I stayed with first Fedora Workstation and now Fedora Silverblue.

I love how up-to-date yet stable and polished everything is in Fedora Linux. For Silverblue, specifically, I love how I can let it update in the background without any fear of things breaking and then reboot into the new updated version later without any waiting, and rollback to a previous deployment or even reset to a pristine state if I break something.

This article will demonstrate how to configure Fedora Server as an alert and notification server that can place calls using an Asterisk PBX and send SMS text messages using Twilio. The use of the SMS message feature is optional. By using the call_only endpoint of the caller_prometheus_webhook component, you can limit the alerts to voice calls only.

Please consider that interacting with the Asterisk PBX is not easy. But it isn’t too hard either. If this is your first time working with this kind of application, coming to understand the concepts may require some patience. Fortunately, Fedora Server can be configured with Ansible and the installation of the py-phone-caller containers will not be difficult.

Note: The py-phone-caller packages do not have any endorsement or relation with Twilio or Asterisk PBX/FreePBX. These services and products were chosen for their ease of use and their commitment to open source.

Note: This guide assumes that the server is on a local area network (LAN) that is behind a firewall configured to block all outside connections. Do not open the services’ ports to the internet without requiring authentication.

The big picture

The py-phone-caller components are represented by the blue boxes. The third party components or dependencies are the green boxes. And the yellow box represents the receiver of the calls and/or text messages.

<figure class="wp-block-image"></figure>

Component overview

• generate_audio

  • Role: used to create and host the audio files player by the Asterisk PBX.
  • Container repository: quay.io/py-phone-caller/generate_audio
  • FROM: fedora:34 (base container image)

• caller_sms

  • Role: used to send the SMS messages through a service provider.
  • Container repository: quay.io/py-phone-caller/caller_sms
  • FROM: redhat/ubi8:8.4-206.1626828523 (base container image)

• caller_prometheus_webhook

  • Role: start a call or send an SMS message when a Prometheus alert is received.
  • Container repository: quay.io/py-phone-caller/caller_prometheus_webhook
  • FROM: redhat/ubi8:8.4-206.1626828523 (base container image)
  • Note: This container has four endpoints that behave differently:

    – call_only: used to originate a single call
    – sms_only: used to send a single SMS
    – sms_before_call: first, send an SMS and later originate a call (after the number of seconds configured in: sms_before_call_wait_seconds)
    – call_and_sms: used to send the SMS and place the call at the same time.

• call_register

  • Role: used to register on the PostgreSQL DB the arriving calls with useful details.
  • Container repository: quay.io/py-phone-caller/call_register
  • FROM: redhat/ubi8:8.4-206.1626828523 (base container image)

• asterisk_ws_monitor

  • Role: this component register the Stasis application against Asterisk and also log the events to the DB (table: asterisk_ws_events).
  • Container repository: quay.io/py-phone-caller/asterisk_ws_monitor
  • FROM: redhat/ubi8:8.4-206.1626828523 (base container image)

• asterisk_recall

  • Role: reading from the database and considering the times_to_dial and seconds_to_forget configuration parameters, retries a failed or not acknowledged call (… press 4 to acknowledge…).
  • Container repository: quay.io/py-phone-caller/asterisk_recall
  • FROM: redhat/ubi8:8.4-206.1626828523 (base container image)

• asterisk_call

  • Role: has the responsibility to place the calls against the Asterisk PBX through the REST interface.
  • Container repository: quay.io/py-phone-caller/asterisk_call
  • FROM: redhat/ubi8:8.4-206.1626828523 (base container image)

Known limitations

Version 0.0.2 of the py-phone-caller application does not have a call traffic controller. Consequently, if multiple events are sent to asterisk_call at the same time, some notifications may be dropped. For reliability, be sure to send no more than one alert at a time from the caller_prometheus_webhook and send only one HTTP POST at a time when using a HTTP client such as curl. To avoid collisions between messages in asterisk_call, do not send events more frequently than the value configured in seconds_to_forget.

Optionally, support for concurrent calls can be enabled by adding more instances of asterisk_call and asterisk_ws_monitor behind an instance of HAProxy. Each additional instance of asterisk_call will require an additional entry in extensions_custom.conf.

To be clear, you need to add one more instance of each of the following for each additional concurrent request that you want to be able to handle:

• asterisk_call has the responsibility of the call initialization using a given custom extension. This extension calls the Stasis application.
• asterisk_ws_monitor registers new instances of the Stasis application. It is referenced in extensions_custom.conf. Be sure to use unique names.
• An entry in extensions_custom.conf is required for exclusively by the asterisk_call instance. This is also where you will reference the Stasis application name configured in asterisk_stasis_app.

  • Note: You can use the same configuration file and override the settings with environment variables. The variables will have priority over the values in the configuration file. The environmental variables use the same names as their corresponding settings, but they must be in upper case. For example, asterisk_stasis_app becomes ASTERISK_STASIS_APP.

Note: chan_pjsip (PJSIP channel) is not yet supported. Support will be added in a future release.

Prerequisites

• A Twillio account is required for sending SMS messages to cell phones (sending SMS messages is optional).
• A SIP trunk, a media gateway or a SIM card inside a device is required for placing calls to the landlines or cell phones.
• An internet connection.

Note: Twilio is the only SMS service provider currently available. Other providers will be added.

Note: If you choose not to send SMS (Short Message Service) messages and not to place calls to paid phone networks, you can still make calls over a SIP (Session Initiation Protocol) or IAX2 (Inter-Asterisk eXchange version 2) extension of the PBX (Private Branch eXchange) to a softphone (software phone) installed on your cell phone or to a physical phone that supports either of these protocols.

Note: You will need to pay a few cents to place calls and send SMS messages to landlines or cell phones. These services are rarely free.

Systems overview

This guide will use the FreePBX Asterisk distribution. It has a web interface that is more user-friendly. The components that initiate calls will be based on Fedora Server 34. Other Fedora Linux variants such as Fedora Workstation or Fedora Cloud Base might work. But all tests and working setups have been based on Fedora Server or CentOS 7 in Docker containers. I am also planning to test this deployment on RHEL 8 and OpenShift/Kubernetes.

• Fedora Server 34

  • IP Address: 192.168.122.104

• FreePBX 15 (CentOS 7 based)

  • SNG7-PBX-64bit-2104-1.iso
  • IP Address: 192.168.122.234

Note: You can use different IP addresses for the Fedora Server and the Asterisk system.

Configuration of the Asterisk PBX

A SIP Trunk is required to place calls to cell phones or landlines. If you choose to use only local extensions and you do not intend to place calls to external phones, you can configure a SIP or IAX2 extension instead.

The py-phone-caller will also require a custom extension and an ARI (Asterisk REST Interface) user.

Configure the SIP Trunk

1. First click Connectivity.
2. Then click Trunks.

<figure class="wp-block-image"></figure>

1. Click + Add Trunk.
2. Click + Add SIP (chan_sip) Trunk.

<figure class="wp-block-image"></figure>

1. Configure the Trunk Name. This name will form part of the value of asterisk_chan_type in the py-phone-caller configuration. For example, if sip-provider is chosen for Trunk Name, the corresponding py-phone-caller configuration value would be SIP/sip-provider.
2. In the Outbound CallerID, you can use any preferred value.
3. Click the sip Settings tab.

<figure class="wp-block-image"></figure>

1. Click the Outgoing tab.
2. Configure the Trunk Name. Use the same name that you chose previously.
3. On the PEER Details section, enter the configuration values required to reach the SIP provider.
4. To save the configuration, click the Submit button.

<figure class="wp-block-image"></figure>

An example PEER Details configuration:

type=peer
auth=md5
username=your-username
fromuser=your-username
secret=your-password
host=sip.provider.com
port=5060
qualify=yes
insecure=very

1. Click Apply Config.

<figure class="wp-block-image"></figure>

1. Wait until the reloading process is done.

<figure class="wp-block-image"></figure>

You should now have a trunk that can be used to place calls to cell and landline phones.

Note: This configuration has a cost depending on the provider or device used to access the public phone network (cell or landline).

Define a custom dial plan

This extension will initiate the call and then pass control to py-phone-caller. The message will be the description of a Prometheus alert. It will be sent from the Alertmanager to the caller_prometheus_webhook.

1. Click Admin.
2. Click Config Edit.

<figure class="wp-block-image"></figure>

1. From the panel on the left, select extensions_custom.conf.
2. Place the text from the custom dial plan below in the Working on extensions_custom.conf text area on the right.

<figure class="wp-block-image"></figure>

1. Click again on Working on extensions_custom.conf to validate the new settings.
2. Click Save.

<figure class="wp-block-image"></figure>

1. Click Apply Config and wait until the configuration reloading process has finished.

<figure class="wp-block-image"></figure>

An example dial plan:

[py-phone-caller]
exten => 3216,1,Noop()
; Greeting message
same => n,Playback(greeting-message)
; Give the control of the ongoing call to the 'py-phone-caller' Stasis application
same => n,Stasis(py-phone-caller)
; Do a get HTTP request against the 'call_register' when the message was played
same => n,Set(RES=${CURL(http://192.168.122.104:8083/heard?asterisk_chan=${CHANNEL(uniqueid)})})
; Play an audio message in order to get the acknowledgement
; in our case we wait for '4'. "see line xx: {IF($[ ${get} = 4 ..."
same => n,Playback(press-4-for-acknowledgement)
same => n,Playback(beep)
same => n,Read(get,"silence/1",,,,2)
; If not digit is provided go to the priority 20 in order to say 'goodbye'
same => n,Set(gotdigit=${ISNULL(${get})})
same => n,GotoIf(${gotdigit}=1?20)
; When '4' is pressed go to the priority 30, update the DB record
; and say Goodbye.
same => e,Playback(vm-goodbye) ; If there's an error, say goodbye
same => n,Set(NOTIFYACK=${IF($[ ${get} = 4]?3:0)})
same => n,Wait(1)
same => n,GotoIf(${NOTIFYACK}=3?30)
same => 20,Set(NOTIFYACK=2)
same => 21,Playback(vm-goodbye)
same => 22,Wait(1)
same => 23,Hangup()
; Do a get HTTP request against the 'call_register' to update the DB record
; when the call was acknowledged.
same => 30,Set(RES=${CURL(http://192.168.122.104:8083/ack?asterisk_chan=${CHANNEL(uniqueid)})})
same => 31,Playback(vm-goodbye)
same => 32,Wait(1)
same => 33,Hangup()
same => n,Playback(vm-goodbye)
same => n,Hangup()

A step-by-step breakdown of the above dial plan:

1. The extension name ([py-phone-caller])
2. The extension number (3216). You can choose a different number. If you use a different number, be sure to use the same number when configuring py-phone-caller.
3. The PBX plays the audio file greeting-message.wav. How to create this file and copy it to the PBX is explained later in this guide.
4. The Stasis(py-phone-caller) line transfers control of the call to py-phone-caller.
5. After py-phone-caller returns, an HTTP GET request is sent to call_register to record that a message was sent.
6. The PBX plays the audio file press-4-for-acknowledgement.wav.
7. The PBX plays a beep to signal to the callee that it is ready to receive input.
8. The Read(get,"silence/1",,,,2) function waits for the callee’s input.
9. If the callee didn’t press the number 4 the call is terminated.

<figure class="wp-block-image"></figure>

Creating a standard Asterisk extension

A SIP or IAX2 extension can be used instead of a SIP Trunk. However, these protocols will only be able to dial out to softphones or VoIP phones. You will not be able to call a PSTN or cell phone using SIP or IAX2. There are no additional costs when using these extensions. They can be configured directly within the PBX. They can be used on the same local network the PBX is connected to. Or they can be routed over the internet. If you choose to route these protocols over the internet, please use a security layer such as TLS and use secure passwords. If you use a softphone installed on your cell phone, be sure that your internet connection has sufficient bandwidth to provide high-quality audio.

Note: When using these alternative protocols, the value of asterisk_chan_type will be SIP instead of SIP/sip-provider. When using these protocols, a SIP Trunk or Media Gateway is not required.

Note: This endpoint can be viewed as the callee. It is the extension or phone number that will be called when a new Prometheus alert or HTTP POST request is sent to asterisk_call. It is also is possible to initiate calls directly from cron jobs and other scripts.

1. Click Applications.

<figure class="wp-block-image"></figure>

1. Click Extensions.

<figure class="wp-block-image"></figure>

1. Click + Add Extension.
2. Click Add New SIP (Legacy) [chan_sip] Extension.

<figure class="wp-block-image"></figure>

1. Configure the User Extension. This is usually a number. This example uses 1614. This number will be used as the value for prometheus_webhook_receivers in the caller_prometheus_webhook configuration block.
2. Configure the Display Name. This text will appear on the display of the callee’s phone.
3. Provide a Secret. This is the password for your softphone or SIP phone. FreePBX will suggest a strong password automatically.
4. Click Submit.

<figure class="wp-block-image"></figure>

1. Click Apply Config.

<figure class="wp-block-image"></figure>

1. Wait for Reloading to finish.

<figure class="wp-block-image"></figure>

Configure the ARI user

The ARI (Asterisk REST Interface) user will be used in asterisk_ws_monitor to register the Stasis application (a permanent WebSocket connection). This account will also be used by asterisk_call to make calls to the Asterisk PBX (or FreePBX).

1. Click Settings.
2. Click Asterisk REST Interface Users.

<figure class="wp-block-image"></figure>

1. Click + Add User.

<figure class="wp-block-image"></figure>

1. Set the REST Interface User Name. This example uses py-phone-caller.
2. Set the REST Interface User Password. You may choose something different from the random value that is suggested.
3. Set the Password Type. During development and testing, it is OK to use Plain Text. In production use Crypt. When using crypt, the password will need to be provided in encrypted form using the sha-512 hash algorithm. See here for more information.
4. Set the Read Only option to No.
5. Click Submit.

<figure class="wp-block-image"></figure>

1. Click Apply Config.

<figure class="wp-block-image"></figure>

1. Wait for Reloading to finish.

<figure class="wp-block-image"></figure>

Create the audio recordings

• Greeting message: “Hello. This is a recorded message from the alerting system.“
• Request acknowledgement: “Please press the number four to acknowledge this call.”

Install espeak.

$ sudo dnf -y install espeak

Generate the audio files in wave format.

$ espeak -s 140 -g 4 -w /tmp/greeting-message_22050.wav "Hello. This is a recorded message from the alerting system."
$ espeak -s 140 -g 4 -w /tmp/press-4-for-acknowledgement_22050.wav "After the beep, please press the number four to acknowledge this call."

The espeak program records the audio files with a sampling frequency of 22050 Hz. However, Asterisk requires 8000 Hz. Install sox and use it to convert the files to the required format.

$ sudo dnf -y install sox
$ sox /tmp/greeting-message_22050.wav -r 8000 -c 1 /tmp/greeting-message.wav
$ sox /tmp/press-4-for-acknowledgement_22050.wav -r 8000 -c 1 /tmp/press-4-for-acknowledgement.wav

You can read more about espeak in its man pages and on Fedora Magazine. Alternatively, if you have a microphone and you know someone with a good voice you can record a real person.

The above example audio files are available here: assets/generic-audio-for-dialplan

Upload the recordings to FreePBX

Upload the audio files you created in the previous section to your Asterisk system.

$ scp /tmp/*.wav root@192.168.122.234:~
root@192.168.122.234's password:

greeting-message.wav                                                                 100%  107KB  21.7MB/s   00:00
press-4-for-acknowledgement.wav                                                      100%   77KB  21.9MB/s   00:00

Login to the Asterisk (FreePBX) system.

$ ssh root@192.168.122.234
root@192.168.122.234's password:
Last failed login: Tue Aug  3 23:29:04 UTC 2021 from 192.168.122.1 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Tue Aug  3 23:33:50 2021 from 192.168.122.1
______                   ______ ______ __   __
|  ___|                  | ___ \| ___ \\ \ / /
| |_    _ __   ___   ___ | |_/ /| |_/ / \ V /
|  _|  | '__| / _ \ / _ \|  __/ | ___ \ /   \
| |    | |   |  __/|  __/| |    | |_/ // /^\ \
\_|    |_|    \___| \___|\_|    \____/ \/   \/

NOTICE! You have 2 notifications! Please log into the UI to see them!
Current Network Configuration
+-----------+-------------------+-------------------------+
| Interface | MAC Address       | IP Addresses            |
+-----------+-------------------+-------------------------+
| eth0      | 52:54:00:F1:1C:F6 | 192.168.122.234         |
|           |                   | fe80::5054:ff:fef5:6cf1 |
+-----------+-------------------+-------------------------+
[...]

[root@freepbx ~]#

This guide uses Asterisk’s default language, English. For English, the audio files are stored in /var/lib/asterisk/sounds/en. Move the uploaded audio files to this directory. If you configured a different language, you would need to move the files to a different directory.

The audio files should be in root’s home directory (/root). Move them to Asterisk’s directory.

[root@freepbx ~]# mv greeting-message.wav press-4-for-acknowledgement.wav /var/lib/asterisk/sounds/en

Finally, set the ownership for the files to asterisk:asterisk.

[root@freepbx ~]# chown asterisk:asterisk /var/lib/asterisk/sounds/en/{greeting-message.wav,press-4-for-acknowledgement.wav}

The configuration of the Asterisk PBX should now be complete.

Install the required software packages

This guide is using Fedora Server edition. Login to your Fedora server and use the sudo command to change to root user.

$ sudo -i

Install Podman.

# dnf -y install podman podman-plugins podman-docker
Last metadata expiration check: 0:42:59 ago on Wed 28 Jul 2021 23:31:23 PM CEST.
Dependencies resolved.
=============================================================================================================================================================================
 Package                                            Architecture                  Version                                               Repository                      Size
=============================================================================================================================================================================
Installing:
 podman                                             x86_64                        3:3.2.3-1.fc34                                        updates                         12 M
 podman-docker                                      noarch                        3:3.2.3-1.fc34                                        updates                        177 k
 podman-plugins                                     x86_64                        3:3.2.3-1.fc34                                        updates                        2.6 M
Installing dependencies:
 conmon                                             x86_64                        2:2.0.29-2.fc34                                       updates                         53 k
 container-selinux                                  noarch                        2:2.164.1-1.git563ba3f.fc34                           updates                         48 k
 containernetworking-plugins                        x86_64                        1.0.0-0.2.rc1.fc34                                    updates                        8.9 M
 containers-common                                  noarch                        4:1-21.fc34                                           updates                         61 k
 criu                                               x86_64                        3.15-3.fc34                                           fedora                         521 k
 criu-libs                                          x86_64                        3.15-3.fc34                                           fedora                          31 k
 crun                                               x86_64                        0.20.1-1.fc34                                         updates                        172 k
 fuse-common                                        x86_64                        3.10.4-1.fc34                                         updates                        8.5 k
 fuse3                                              x86_64                        3.10.4-1.fc34                                         updates                         54 k
 fuse3-libs                                         x86_64                        3.10.4-1.fc34                                         updates                         91 k
 libbsd                                             x86_64                        0.10.0-7.fc34                                         fedora                         106 k
 libnet                                             x86_64                        1.2-2.fc34                                            fedora                          58 k
 libslirp                                           x86_64                        4.4.0-4.fc34                                          updates                         68 k
 yajl                                               x86_64                        2.1.0-16.fc34                                         fedora                          38 k
Installing weak dependencies:
 catatonit                                          x86_64                        0.1.5-4.fc34                                          fedora                         305 k
 fuse-overlayfs                                     x86_64                        1.5.0-1.fc34                                          fedora                          75 k
 slirp4netns                                        x86_64                        1.1.9-1.fc34                                          fedora                          57 k

Transaction Summary
=============================================================================================================================================================================
Install  20 Packages

Total download size: 25 M
Installed size: 123 M
[...]

Install Ansible and PostgreSQL.

# dnf install -y ansible python3-psycopg2 postgresql
Last metadata expiration check: 0:38:54 ago on Wed 28 Jul 2021 23:41:48 PM CEST.
Dependencies resolved.
=============================================================================================================================================================================
 Package                                           Architecture                       Version                                      Repository                           Size
=============================================================================================================================================================================
Installing:
 ansible                                           noarch                             2.9.23-1.fc34                                updates                              15 M
 python3-psycopg2                                  x86_64                             2.8.6-3.fc34                                 fedora                              183 k
Installing dependencies:
 libpq                                             x86_64                             13.3-1.fc34                                  updates                             202 k
 libsodium                                         x86_64                             1.0.18-7.fc34                                fedora                              165 k
 python3-babel                                     noarch                             2.9.1-1.fc34                                 updates                             5.8 M
 python3-bcrypt                                    x86_64                             3.1.7-7.fc34                                 fedora                               44 k
 python3-cffi                                      x86_64                             1.14.5-1.fc34                                fedora                              244 k
 python3-chardet                                   noarch                             4.0.0-1.fc34                                 fedora                              214 k
 python3-cryptography                              x86_64                             3.4.6-1.fc34                                 fedora                              1.4 M
 python3-idna                                      noarch                             2.10-3.fc34                                  fedora                               99 k
 python3-jinja2                                    noarch                             2.11.3-1.fc34                                fedora                              493 k
 python3-jmespath                                  noarch                             0.10.0-1.fc34                                updates                              46 k
 python3-markupsafe                                x86_64                             1.1.1-10.fc34                                fedora                               32 k
 python3-ntlm-auth                                 noarch                             1.5.0-2.fc34                                 fedora                               53 k
 python3-ply                                       noarch                             3.11-11.fc34                                 fedora                              103 k
 python3-pycparser                                 noarch                             2.20-3.fc34                                  fedora                              126 k
 python3-pynacl                                    x86_64                             1.4.0-2.fc34                                 fedora                              110 k
 python3-pysocks                                   noarch                             1.7.1-8.fc34                                 fedora                               35 k
 python3-pytz                                      noarch                             2021.1-2.fc34                                fedora                               49 k
 python3-pyyaml                                    x86_64                             5.4.1-2.fc34                                 fedora                              194 k
 python3-requests                                  noarch                             2.25.1-1.fc34                                fedora                              114 k
 python3-requests_ntlm                             noarch                             1.1.0-14.fc34                                fedora                               18 k
 python3-urllib3                                   noarch                             1.25.10-5.fc34                               updates                             174 k
 python3-xmltodict                                 noarch                             0.12.0-11.fc34                               fedora                               23 k
 sshpass                                           x86_64                             1.09-1.fc34                                  fedora                               27 k
Installing weak dependencies:
 python3-paramiko                                  noarch                             2.7.2-4.fc34                                 fedora                              287 k
 python3-pyasn1                                    noarch                             0.4.8-4.fc34                                 fedora                              133 k
 python3-winrm                                     noarch                             0.4.1-2.fc34                                 fedora                               79 k

Transaction Summary
=============================================================================================================================================================================
Install  28 Packages

Total download size: 25 M
Installed size: 144 M

[...]

Exit the root shell.

# exit
logout

Install the Ansible role for managing Podman.

$ ansible-galaxy collection install containers.podman
Process install dependency map
Starting collection install process
Installing 'containers.podman:1.6.1' to '/home/fedora/.ansible/collections/ansible_collections/containers/podman'

Install the Ansible role for managing PostgreSQL.

$ ansible-galaxy collection install community.postgresql
Process install dependency map
Starting collection install process
Installing 'community.postgresql:1.4.0' to '/home/fedora/.ansible/collections/ansible_collections/community/postgresql'

Verify the Podman installation by running a small test container.

$ podman run hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob b8dfde127a29 done
Copying config d1165f2212 done
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

The following files will be required to install py-phone-caller via Ansible.

• caller_config.toml.jinja2
• py-phone-caller-podman.yml
• py_phone_caller_vars_file.yml

Fisrt, create a directory to store the installation playbook and cd into it.

$ mkdir ansible_py-phone-caller
$ cd ansible_py-phone-caller

Now download the files.

$ wget https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/caller_config.toml.jinja2
--2021-07-28 23:48:11--  https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/caller_config.toml.jinja2
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3730 (3.6K) [text/plain]
Saving to: ‘caller_config.toml.jinja2’

caller_config.toml.jinja2                   100%[========================================================================================>]   3.64K  --.-KB/s    in 0s

2021-07-28 23:48:11 (20.4 MB/s) - ‘caller_config.toml.jinja2’ saved [3730/3730]
$ wget https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py-phone-caller-podman.yml
--2021-07-28 23:49:55--  https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py-phone-caller-podman.yml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10793 (11K) [text/plain]
Saving to: ‘py-phone-caller-podman.yml’

py-phone-caller-podman.yml                  100%[========================================================================================>]  10.54K  --.-KB/s    in 0s

2021-07-28 23:49:55 (25.7 MB/s) - ‘py-phone-caller-podman.yml’ saved [10793/10793]
$ wget https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py_phone_caller_vars_file.yml
--2021-07-28 23:51:20--  https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/ansible/rh/py_phone_caller_vars_file.yml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7149 (7.0K) [text/plain]
Saving to: ‘py_phone_caller_vars_file.yml’

py_phone_caller_vars_file.yml               100%[========================================================================================>]   6.98K  --.-KB/s    in 0s

2021-07-28 23:51:20 (14.6 MB/s) - ‘py_phone_caller_vars_file.yml’ saved [7149/7149]

The variables file py_phone_caller_vars_file.yml:

---
# Variables files for the 'py-phone-caller' installed through Podman

ansible_python_interpreter: /usr/bin/python3

# Podman / 'py-phone-caller' vars
container_host: 192.168.122.104
installation_user: fedora
installation_folder: "/home/{{ installation_user }}"
installation_folder_name: py-phone-caller
caller_config_toml_template_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/9c0bb97110cef988adfd23a6c0cb8e25168bbdfc/assets/ansible/rh/caller_config.toml.jinja2"
py_phone_caller_config_tmp_path: /tmp/caller_config.toml.jinja
py_phone_caller_config_path: "{{ installation_folder }}/{{ installation_folder_name }}/config/caller_config.toml"
config_mounted_in_container: /opt/py-phone-caller/config/caller_config.toml:Z
py_phone_caller_version: 0.0.2
container_registry_url: quay.io/py-phone-caller
py_phone_caller_network: py-phone-caller
py_phone_caller_subnet: 172.19.0.0/24
asterisk_ws_monitor_ip: 172.19.0.10
asterisk_recall_ip: 172.19.0.11
postgres_ip: 172.19.0.50
generate_audio_ip: 172.19.0.82
call_register_ip: 172.19.0.83
asterisk_call_ip: 172.19.0.81
caller_prometheus_webhook_ip: 172.19.0.84
caller_sms_ip: 172.19.0.85

# PostgreSQL container vars
db_schema_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/DB/db-schema.sql"
postgresql_container_image: "docker.io/library/postgres:13.3-alpine3.14"
postgresql_login_host: 127.0.0.1
postgresql_admin: postgres
postgresql_admin_pass: Use-A-Secure-Password-Here

# SystemD user integration
container_asterisk_call_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call.service"
container_asterisk_call_register_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call_register.service"
container_asterisk_recall_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_recall.service"
container_asterisk_ws_monitor_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_ws_monitor.service"
container_caller_prometheus_webhook_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_prometheus_webhook.service"
container_caller_sms_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_sms.service"
container_generate_audio_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-generate_audio.service"
container_postgres_13_service_url: "https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-postgres_13.service"

systemd_user_path: "/home/{{ installation_user }}/.config/systemd/user"
container_asterisk_call_service_unit: "container-asterisk_call.service"
container_asterisk_call_register_service_unit: "container-asterisk_call_register.service"
container_asterisk_recall_service_unit: "container-asterisk_recall.service"
container_asterisk_ws_monitor_service_unit: "container-asterisk_ws_monitor.service"
container_caller_prometheus_webhook_service_unit: "container-caller_prometheus_webhook.service"
container_caller_sms_service_unit: "container-caller_sms.service"
container_generate_audio_service_unit: "container-generate_audio.service"
container_postgres_13_service_unit: "container-postgres_13.service"

# py-phone-caller - 'caller_config.toml' vars
# [commons]
asterisk_user: "py-phone-caller"
asterisk_pass: "Use-A-Secure-Password-Here"
asterisk_host: "192.168.122.234"
asterisk_web_port: "8088"
asterisk_http_scheme: "http"

# [asterisk_call]
asterisk_ari_channels: "ari/channels"
asterisk_ari_play: "play?media=sound"
asterisk_context: "py-phone-caller"
asterisk_extension: "3216"
asterisk_chan_type: "SIP/sip-provider"
asterisk_callerid: "Py-Phone-Caller"
asterisk_call_http_scheme: "http"
asterisk_call_host: "{{ container_host }}"
asterisk_call_port: "8081"
asterisk_call_app_route_asterisk_init: "asterisk_init"
asterisk_call_app_route_play: "play"
seconds_to_forget: 300
client_timeout_total: 5 # For 'ClientTimeout(total=5)'

# [call_register]
call_register_http_scheme: "http"
call_register_host: "{{ container_host }}"
call_register_port: "8083"
call_register_app_route_register_call: "register_call"
call_register_app_route_voice_message: "msg"
call_register_app_route_acknowledge: "ack"
call_register_app_route_heard: "heard"

# [asterisk_ws_monitor]
asterisk_stasis_app: "py-phone-caller"

# [asterisk_recall]
times_to_dial: 3

# [generate_audio]
generate_audio_http_scheme: "http"
generate_audio_host: "{{ container_host }}"
generate_audio_port: "8082"
generate_audio_app_route: "make_audio"
gcloud_tts_language_code: "es"
serving_audio_folder: "audio"
num_of_cpus: 4

# [caller_prometheus_webhook]
prometheus_webhook_port: "8084"
prometheus_webhook_app_route_call_only: "call_only"
prometheus_webhook_app_route_sms_only: "sms_only"
prometheus_webhook_app_route_sms_before_call: "sms_before_call"
prometheus_webhook_app_route_call_and_sms: "call_and_sms"
prometheus_webhook_receivers: '[ "+123456789" ]'

# [caller_sms]
caller_sms_http_scheme: "http"
caller_sms_host: "{{ container_host }}"
caller_sms_port: "8085"
caller_sms_app_route: "send_sms"
sms_before_call_wait_seconds: 120
caller_sms_carrier: "twilio"
twilio_account_sid: "Your-Twilio-account-sid"
twilio_auth_token: "Your-Twilio-auth-token"
twilio_sms_from: "+1987654321"

# [database]
db_host: "{{ postgres_ip }}"
db_name: "py_phone_caller"
db_user: "py_phone_caller"
db_password: 'Use-A-Secure-Password-Here'
db_max_size: 50
db_max_inactive_connection_lifetime: 30.0

# [logger]
log_formatter: "%(asctime)s %(message)s"
acknowledge_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/ack?asterisk_chan=[The Asterisk Channel ID]"
heard_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/heard?asterisk_chan=[The Asterisk Channel ID]"
registercall_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/?phone=[Destination Phone Number]&messagge=[Alert Message Text]&asterisk_chan=[The Asterisk Channel ID]"
voice_message_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/msg?asterisk_chan=[The Asterisk Channel ID]"
asterisk_call_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/asterisk?phone=[Destination Phone Number]&messagge=[Alert Message Text]"
asterisk_play_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/play?asterisk_chan=[The Asterisk Channel ID]&msg_chk_sum=[The message cecksum]"
generate_audio_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/make_audio?messagge=[Alert Message Text]&msg_chk_sum=[The message cecksum]"
caller_sms_error: "Lost parameter, Usage: Method: POST - http://ADDRESS/?phone=[Destination Phone Number]&messagge=[Alert Message Text]"
lost_directory_error: "The folder to serve the audio files was not found."

Install and configuring py-phone-caller using Ansible.

Ansible is a really cool configuration management and orchestration tool. Using Ansible, all the py-phone-caller components can be installed and configured automatically. After all the parameters are set in the assets/ansible/rh/py_phone_caller_vars_file.yml file, run ansible-playbook and in a few minutes you will have a fully configured and ready to be used py-phone-caller.

Variables that you might want to change in the assets/ansible/rh/py_phone_caller_vars_file.yml file:

[...]
# Podman / 'py-phone-caller' vars
[...]
container_host: 192.168.122.104 # Here we need to configure the IP address of our Fedora Server instance
installation_user: fedora # The user within the installation will be done.

[...]

# PostgreSQL container vars
[...]
postgresql_admin_pass: Use-A-Secure-Password-Here # The administrative password for PostgreSQL (user: postgres)

[...]
# py-phone-caller - 'caller_config.toml' vars
# [commons]
asterisk_pass: "Use-A-Secure-Password-Here" # The password for the 'py-phone-caller' Asterisk ARI user
asterisk_host: "192.168.122.234" # The IP address of the Asterisk (in our case FreePBX) instance.
[...]

# [caller_sms]
# Only if we want send SMS through Twilio
[...]
twilio_account_sid: "Your-Twilio-account-sid" # The Twilio SID
twilio_auth_token: "Your-Twilio-auth-token" # The Twilio auth token
twilio_sms_from: "+1987654321" # The Twilio number
[...]

# [database]
[...]
db_password: 'Use-A-Secure-Password-Here' # The password for the PostgreSQL 'py-phone-caller' user.
[...]

To install py-phone-caller, run ansible-playbook.

$ ansible-playbook --connection=local --limit=127.0.0.1 --inventory=127.0.0.1, ansible_py-phone-caller/py-phone-caller-podman.yml

PLAY [Configuring 'py-phone-caller' installed through 'Podman'] *************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [Creating the 'py-phone-caller' directory tree at '/home/fedora/py-phone-caller'] **************************************************************************************
changed: [127.0.0.1]

TASK [Check that the 'caller_config.toml' exists] ***************************************************************************************************************************
ok: [127.0.0.1]

TASK [Download the 'caller_config.toml' template file] **********************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the configuration file 'caller_config.toml' through a template] **********************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'py-phone-caller' container network] *********************************************************************************************************************
changed: [127.0.0.1]

TASK [Checking if the 'pgdata' volume exists] *******************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["podman", "volume", "inspect", "pgdata"], "delta": "0:00:00.059469", "end": "2021-07-28 22:04:44.174686", "msg": "non-zero return code", "rc": 125, "start": "2021-07-28 22:04:44.115217", "stderr": "Error: error inspecting object: no such volume pgdata", "stderr_lines": ["Error: error inspecting object: no such volume pgdata"], "stdout": "[]", "stdout_lines": ["[]"]}

TASK [Creating the 'pgdata' volume for the PostgreSQL container] ************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'postgres_13' container] *********************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'asterisk_call' container] *******************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'caller_prometheus_webhook' container] *******************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'caller_sms' container] **********************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'generate_audio' container] ******************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'py_phone_caller' PostgreSQL user] ***********************************************************************************************************************
changed: [127.0.0.1]

TASK [Gattering info about the 'py_phone_caller' DB (if exists)] ************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "unable to connect to database: FATAL:  database \"py_phone_caller\" does not exist\n"}

TASK [Download the dumped 'py_phone_caller' DB schema (SQL format)] *********************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'py_phone_caller' DB] ************************************************************************************************************************************
changed: [127.0.0.1]

TASK [Restoring the 'py_phone_caller' DB by restoring a dump of the schema] *************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'asterisk_ws_monitor' container] *************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'asterisk_recall' container] *****************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the 'call_register' container] *******************************************************************************************************************************
changed: [127.0.0.1]

TASK [Creating the Systemd user folder] *************************************************************************************************************************************
changed: [127.0.0.1]

TASK [Download the Systemd Unit files] **************************************************************************************************************************************
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call.service', 'path': 'container-asterisk_call.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_call_register.service', 'path': 'container-asterisk_call_register.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_recall.service', 'path': 'container-asterisk_recall.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-asterisk_ws_monitor.service', 'path': 'container-asterisk_ws_monitor.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_prometheus_webhook.service', 'path': 'container-caller_prometheus_webhook.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-caller_sms.service', 'path': 'container-caller_sms.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-generate_audio.service', 'path': 'container-generate_audio.service'})
changed: [127.0.0.1] => (item={'url': 'https://raw.githubusercontent.com/jcfdeb/py-phone-caller/main/assets/systemd-units/as-non-root-container/container-postgres_13.service', 'path': 'container-postgres_13.service'})

TASK [Enable and run the user SystemD services for 'py-phone-caller'] *******************************************************************************************************
changed: [127.0.0.1] => (item=container-asterisk_call.service)
changed: [127.0.0.1] => (item=container-asterisk_call_register.service)
changed: [127.0.0.1] => (item=container-asterisk_recall.service)
changed: [127.0.0.1] => (item=container-asterisk_ws_monitor.service)
changed: [127.0.0.1] => (item=container-caller_prometheus_webhook.service)
changed: [127.0.0.1] => (item=container-caller_sms.service)
changed: [127.0.0.1] => (item=container-generate_audio.service)
changed: [127.0.0.1] => (item=container-postgres_13.service)

TASK [Deleting the '/tmp' files] ********************************************************************************************************************************************
changed: [127.0.0.1] => (item=/tmp/db-schema.sql)
changed: [127.0.0.1] => (item=/tmp/caller_config.toml.jinja)

PLAY RECAP ******************************************************************************************************************************************************************
127.0.0.1                  : ok=23   changed=21   unreachable=0    failed=0    skipped=0    rescued=2    ignored=0

Run the following Firewalld commands to allow the necessary connections.

$ sudo firewall-cmd --add-source="192.168.122.0/24" --permanent
success
$ sudo firewall-cmd --add-source="172.19.0.0/24" --permanent
success
$ sudo firewall-cmd --add-port=8081/tcp --permanent
success
$ sudo firewall-cmd --add-port=8082/tcp --permanent
success
$ sudo firewall-cmd --add-port=8083/tcp --permanent
success
$ sudo firewall-cmd --add-port=8084/tcp --permanent
success
$ sudo firewall-cmd --reload
success

Install and Configure the Prometheus Monitoring Stack

The final pieces to be installed are the Node Expoter, the Alertmanager and Prometheus. With these, you will be able to monitor, alert and collect metrics when some metric violates a condition defined in the alerting rules.

Packages to be installed:

• golang-github-prometheus
• golang-github-prometheus-alertmanager
• golang-github-prometheus-node-exporter

Enter the following command to install the packages.

$ sudo dnf -y install golang-github-prometheus golang-github-prometheus-alertmanager golang-github-prometheus-node-exporter

Last metadata expiration check: 0:17:09 ago on Thu 12 Aug 2021 21:23:21 PM CEST.
Dependencies resolved.
=============================================================================================================================================================================
 Package                                                        Architecture                   Version                                 Repository                       Size
=============================================================================================================================================================================
Installing:
 golang-github-prometheus                                       x86_64                         2.24.1-6.fc34                           updates                          31 M
 golang-github-prometheus-alertmanager                          x86_64                         0.21.0-3.fc34                           fedora                           13 M
 golang-github-prometheus-node-exporter                         x86_64                         1.1.1-2.fc34                            fedora                          4.5 M

Transaction Summary
=============================================================================================================================================================================
Install  3 Packages

Total download size: 48 M
Installed size: 212 M
Downloading Packages:
[...]

A quick illustration of how the components interact

Node Exporter <-- Prometheus --> Alertmanager --> caller_prometheus_webhook

The Node Exporter exposes the metrics of the Fedora Server system. Prometheus does a periodic pull of these metrics. If some alerting rules are defined and one or more metrics violates a condition then Prometheus will send a request to the Alertmanager. Alertmanager will then call caller_prometheus_webhook.

The Prometheus Alertmanager

The role of this component is to trigger the different notification systems (e-mail, PagerDuty, Slack, VictorOps, etc.). It can also make a POST request to a webhook receiver such as caller_prometheus_webhook to place a call or send an SMS message (or whatever else you have configured it to do).

One useful feature of this component is the ability to silence the alerts from the web interface. When someone takes an action to resolve the problem, they can silence it. Otherwise the default configuration would repeat the alert in three hours (repeat_interval: 3h).

A list of the package contents:

# dnf repoquery -l golang-github-prometheus-alertmanager
Last metadata expiration check: 0:50:29 ago on Thu 12 Aug 2021 04:23:21 PM CEST.
/usr/bin/alertmanager
/usr/bin/amtool
/usr/lib/.build-id
/usr/lib/.build-id/0b
/usr/lib/.build-id/0b/275747139b1a6f1b398166b54ee83f625e1d4e
/usr/lib/.build-id/4d
/usr/lib/.build-id/4d/be93dda0226df14625b47ea1a6747eacd1e0d1
/usr/share/doc/golang-github-prometheus-alertmanager
/usr/share/doc/golang-github-prometheus-alertmanager/CHANGELOG.md
/usr/share/doc/golang-github-prometheus-alertmanager/MAINTAINERS.md
/usr/share/doc/golang-github-prometheus-alertmanager/README.md
/usr/share/doc/golang-github-prometheus-alertmanager/doc
/usr/share/doc/golang-github-prometheus-alertmanager/doc/arch.svg
/usr/share/doc/golang-github-prometheus-alertmanager/doc/arch.xml
/usr/share/doc/golang-github-prometheus-alertmanager/doc/design
/usr/share/doc/golang-github-prometheus-alertmanager/doc/design/secure-cluster-traffic.md
/usr/share/doc/golang-github-prometheus-alertmanager/doc/examples
/usr/share/doc/golang-github-prometheus-alertmanager/doc/examples/simple.yml
/usr/share/doc/golang-github-prometheus-alertmanager/examples
/usr/share/doc/golang-github-prometheus-alertmanager/examples/ha
/usr/share/doc/golang-github-prometheus-alertmanager/examples/ha/alertmanager.yml
/usr/share/doc/golang-github-prometheus-alertmanager/examples/ha/send_alerts.sh
/usr/share/doc/golang-github-prometheus-alertmanager/examples/webhook
/usr/share/doc/golang-github-prometheus-alertmanager/examples/webhook/echo.go
/usr/share/licenses/golang-github-prometheus-alertmanager
/usr/share/licenses/golang-github-prometheus-alertmanager/COPYRIGHT.txt
/usr/share/licenses/golang-github-prometheus-alertmanager/LICENSE
/usr/share/licenses/golang-github-prometheus-alertmanager/NOTICE

From the previous code snippet you can see that there is no systemd service file, directory to store the data, or configuration directory. You will need to create them manually.

# mkdir -p /etc/alertmanager/template /var/lib/prometheus/alertmanager/data
# chown -R prometheus. /var/lib/prometheus/alertmanager
# cp /usr/share/doc/golang-github-prometheus-alertmanager/doc/examples/simple.yml /etc/alertmanager/alertmanager.yml

Next, add the following configuration blocks to /etc/alertmanager/alertmanager.yml.

In the routes section:

[...]
  # py-phone-caller example
  - match:
      severity: disaster
    receiver: py-phone-caller
[...]

In the receivers section:

[...]
- name: 'py-phone-caller'
  webhook_configs:
  # Onr of the endpoints of 'caller_prometheus_webhook'
  - url: 'http://127.0.0.1:8084/sms_before_call'
    send_resolved: false
[...]

The complete /etc/alertmanager/alertmanager.yml file:

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

# The directory from which notification templates are read.
templates:
- '/etc/alertmanager/template/*.tmpl'

# The root route on which each incoming alert enters.
route:
  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  #
  # To aggregate by all possible labels use '...' as the sole label name.
  # This effectively disables aggregation entirely, passing through all
  # alerts as-is. This is unlikely to be what you want, unless you have
  # a very low alert volume or your upstream notification system performs
  # its own grouping. Example: group_by: [...]
  group_by: ['alertname', 'cluster', 'service']

  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 30s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 3h

  # A default receiver
  receiver: team-X-mails

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.

  # The child route trees.
  routes:
  # This routes performs a regular expression match on alert labels to
  # catch alerts that are related to a list of services.
  - match_re:
      service: ^(foo1|foo2|baz)$
    receiver: team-X-mails
    # The service has a sub-route for critical alerts, any alerts
    # that do not match, i.e. severity != critical, fall-back to the
    # parent node and are sent to 'team-X-mails'
    routes:
    - match:
        severity: critical
      receiver: team-X-pager
  - match:
      service: files
    receiver: team-Y-mails


  # py-phone-caller example
  - match:
      severity: disaster
    receiver: py-phone-caller


    routes:
    - match:
        severity: critical
      receiver: team-Y-pager

  # This route handles all alerts coming from a database service. If there's
  # no team to handle it, it defaults to the DB team.
  - match:
      service: database
    receiver: team-DB-pager
    # Also group alerts by affected database.
    group_by: [alertname, cluster, database]
    routes:
    - match:
        owner: team-X
      receiver: team-X-pager
      continue: true
    - match:
        owner: team-Y
      receiver: team-Y-pager


# Inhibition rules allow to mute a set of alerts given that another alert is
# firing.
# We use this to mute any warning-level notifications if the same alert is
# already critical.
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  # Apply inhibition if the alertname is the same.
  # CAUTION:
  #   If all label names listed in `equal` are missing
  #   from both the source and target alerts,
  #   the inhibition rule will apply!
  equal: ['alertname', 'cluster', 'service']


receivers:
- name: 'team-X-mails'
  email_configs:
  - to: 'team-X+alerts@example.org'

- name: 'team-X-pager'
  email_configs:
  - to: 'team-X+alerts-critical@example.org'
  pagerduty_configs:
  - service_key: <team-X-key>

- name: 'team-Y-mails'
  email_configs:
  - to: 'team-Y+alerts@example.org'

- name: 'team-Y-pager'
  pagerduty_configs:
  - service_key: <team-Y-key>

- name: 'team-DB-pager'
  pagerduty_configs:
  - service_key: <team-DB-key>

- name: 'py-phone-caller'
  webhook_configs:
  # Onr of the endpoints of 'caller_prometheus_webhook'
  - url: 'http://127.0.0.1:8084/sms_before_call'
    send_resolved: false

Note: This is an example file and several blocks can be removed because they aren’t used. For your use case it will work fine. The unused parts weren’t removed in order to show a complete landscape of this file.

Run systemctl cat prometheus.service to view the prometheus.service systemd unit file.

# systemctl cat prometheus.service
# /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus service monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Wants=network-online.target
After=network-online.target

[Service]
Restart=on-failure
EnvironmentFile=/etc/sysconfig/prometheus
User=prometheus
Group=prometheus
ExecStart=/usr/bin/prometheus \
          --config.file=${CONFIG_FILE} \
          --storage.tsdb.path=${STORAGE_TSDB_PATH} \
          --web.console.libraries=${WEB_CONSOLE_LIBRARIES_PATH} \
          --web.console.templates=${WEB_CONSOLE_TEMPLATES_PATH} \
          --web.listen-address=${WEB_LISTEN_ADDRESS}
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

The /etc/systemd/system/alertmanager.service file:

[Unit]
Description=Alertmanager for the Prometheus monitoring system
Documentation=https://prometheus.io/docs/alerting/latest/alertmanager/
Wants=network-online.target
After=network-online.target

[Service]
Restart=on-failure
User=prometheus
Group=prometheus
ExecStart=/usr/bin/alertmanager --config.file="/etc/alertmanager/alertmanager.yml" --storage.path="/var/lib/prometheus/alertmanager/data"
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

Enable alertmanager.service.

# systemctl enable --now alertmanager.service
Created symlink /etc/systemd/system/multi-user.target.wants/alertmanager.service → /etc/systemd/system/alertmanager.service.

Use the following command to verify that the service is running.

# systemctl status alertmanager.service
● alertmanager.service - Alertmanager for the Prometheus monitoring system
     Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2021-08-12 21:44:32 CEST; 1s ago
       Docs: https://prometheus.io/docs/alerting/latest/alertmanager/
   Main PID: 49281 (alertmanager)
      Tasks: 11 (limit: 4647)
     Memory: 16.2M
        CPU: 86ms
     CGroup: /system.slice/alertmanager.service
             └─49281 /usr/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/prometheus/alertmanager/data

Aug 12 21:44:32 fedora systemd[1]: Started Alertmanager for the Prometheus monitoring system.
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.390Z caller=main.go:216 msg="Starting Alertmanager" version="(version=, branch=, revision=)"
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.390Z caller=main.go:217 build_context="(go=go1.16rc1, user=, date=)"
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.391Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=1>
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.392Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.425Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/>
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.426Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration>
Aug 12 21:44:32 fedora alertmanager[49281]: level=info ts=2021-08-12T19:44:32.430Z caller=main.go:485 msg=Listening address=:9093

The Node Exporter

This component exposes various metrics on the host where it is running. For more information about the metrics exposed by this component, see the README.md file in the Github repository.

Enable and start the node_exporter service.

# systemctl enable --now node_exporter.service
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /etc/systemd/system/node_exporter.service.

Check the status of the node_exporter service.

# systemctl status node_exporter.service
● node_exporter.service - Node Exporter
     Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2021-08-12 21:55:48 CEST; 7s ago
   Main PID: 49401 (node_exporter)
      Tasks: 6 (limit: 4647)
     Memory: 5.3M
        CPU: 15ms
     CGroup: /system.slice/node_exporter.service
             └─49401 /usr/sbin/node_exporter --collector.textfile.directory /var/lib/node_exporter/textfile_collector

Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=thermal_zone
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=time
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=timex
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=udp_queues
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=uname
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=vmstat
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=xfs
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:113 collector=zfs
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.066Z caller=node_exporter.go:195 msg="Listening on" address=:9100
Aug 12 21:55:48 fedora node_exporter[49401]: level=info ts=2021-08-12T19:55:48.067Z caller=tls_config.go:191 msg="TLS is disabled." http2=false

Prometheus

You will need to make some changes before starting the Prometheus service. You need to change the default listening port from 9090 to another value. Port 9090 is reserved by Cockpit on Fedora Server edition.

# ss -wenotulipas | grep :9090
tcp   LISTEN 0      4096                        *:9090                *:*     users:(("systemd",pid=1,fd=139)) ino:18756 sk:e cgroup:/system.slice/cockpit.socket v6only:0 <-

The /etc/sysconfig/prometheus file:

CONFIG_FILE=/etc/prometheus/prometheus.yml
STORAGE_TSDB_PATH=/var/lib/prometheus
WEB_CONSOLE_LIBRARIES_PATH=/etc/prometheus/console_libraries
WEB_CONSOLE_TEMPLATES_PATH=/etc/prometheus/consoles
WEB_LISTEN_ADDRESS=127.0.0.1:9091

In the /etc/prometheus/prometheus.yml file, modify are the following blocks.

The alertmanagers block:

[...]
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 127.0.0.1:9093
[...]

The scrape_configs block:

[...]
  # The local node exporter
  - job_name: 'node-exporter'

    static_configs:
    - targets: ['localhost:9100']
[...]

The rule_files block:

[...]
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - "alert_rules.yml"
[...]

The /etc/prometheus/prometheus.yml file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 127.0.0.1:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - "alert_rules.yml"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9091']

  # The local node exporter
  - job_name: 'node-exporter'

    static_configs:
    - targets: ['localhost:9100']

The /etc/prometheus/alert_rules.yml file:

groups:
- name: py-phone-caller test alerts
  rules:

  - alert: NodeExporterDown
    expr: up{job="node-exporter"} == 0
    for: 5m
    labels:
      severity: disaster
    annotations:
      summary: "The Node Exporter instance is down"
      description: "The Node Exporter instance is down, please check soon as possible"

Enable and start the prometheus service

# systemctl enable --now prometheus.service
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /usr/lib/systemd/system/prometheus.service.

Check the status of the prometheus service.

# systemctl status prometheus.service
● prometheus.service - Prometheus service monitoring system and time series database
     Loaded: loaded (/usr/lib/systemd/system/prometheus.service; enabled; vendor preset: disabled)
     Active: active (running) since Thu 2021-08-12 22:03:57 CEST; 6s ago
       Docs: https://prometheus.io/docs/introduction/overview/
             man:prometheus(1)
   Main PID: 49560 (prometheus)
      Tasks: 12 (limit: 4647)
     Memory: 31.9M
        CPU: 102ms
     CGroup: /system.slice/prometheus.service
             └─49560 /usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.libraries=/etc/prometheus/consol>

Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.753Z caller=head.go:645 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.753Z caller=head.go:659 component=tsdb msg="On-disk memory mappable chunks replay completed" dur>
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.754Z caller=head.go:665 component=tsdb msg="Replaying WAL, this may take a while"
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.754Z caller=head.go:717 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.754Z caller=head.go:722 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.>
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.755Z caller=main.go:758 fs_type=XFS_SUPER_MAGIC
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.755Z caller=main.go:761 msg="TSDB started"
Aug 12 22:03:57 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:57.756Z caller=main.go:887 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Aug 12 22:03:58 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:58.005Z caller=main.go:918 msg="Completed loading of configuration file" filename=/etc/prometheus/p>
Aug 12 22:03:58 fedora prometheus[49560]: level=info ts=2021-08-12T20:03:58.006Z caller=main.go:710 msg="Server is ready to receive web requests."

Add the following firewall rules for Prometeus and the Alertmanager.

$ sudo firewall-cmd --add-port=9091/tcp --permanent
success
$ sudo firewall-cmd --add-port=9093/tcp --permanent
success
$ sudo firewall-cmd --reload
success

Behind the scenes

As always, the logs are useful in order to understand what’s happening with your programs.

• Logs of caller_prometheus_webhook

When an alerts is sent from Prometheus to the Alermanager, then the Alertmanager sends an HTTP POST request to the caller_prometheus_webhook. In this example we’ve used the /sms_before_call endpoint and as consequence an SMS will be sent soon as possible and later, the phone call.

2021-08-12 18:45:01,803 Call/Message 'The Node Exporter instance is down, please check soon as possible.' for '+393312345678' through the endpoint 'sms_before_call'
2021-08-12 18:45:01,807 - 172.19.0.84 [12/Aug/2021:16:45:01 +0000] "POST /sms_before_call HTTP/1.1" 200 180 "-" "Alertmanager/"

• Logs of caller_sms

When the caller_prometheus_webhook receives the POST request from the Alertmanager it does other POST requesto to the caller_sms to send the message to the contact configured in prometheus_webhook_receivers. Through the SMS service provider.

2021-08-12 18:45:01,812 Sending the SMS message 'The Node Exporter instance is down, please check soon as possible.' to '+393312345678'
2021-08-12 18:45:01,862 - POST Request: https://api.twilio.com/2010-04-01/Accounts/C3RTyd674gtalirS567q25687g57980gtr53/Messages.json
2021-08-12 18:45:01,862 - PAYLOAD: {'To': '+393312345678', 'From': '+15596123456', 'Body': 'The Node Exporter instance is down, please check soon as possible.'}
2021-08-12 18:45:02,543 - POST Response: 201 {"sid": "ABcDEFGhIJKLMNoPqRsTU1234565671z", "date_created": "Thu, 12 Aug 2021 18:45:02 +0000", "date_updated": "Thu, 12 Aug 2021 18:45:02 +0000", "date_sent": null,
                               "account_sid": "C3RTyd674gtalirS567q25687g57980gtr53", "to": "+393312345678", "from": "+15596123456", "messaging_service_sid": null, "body": "The Node Exporter instance is down, please check soon as possible.",
                               "status": "queued", "num_segments": "1", "num_media": "0", "direction": "outbound-api", "api_version": "2010-04-01", "price": null, "price_unit": "USD", "error_code": null, "error_message": null,
                               "uri": "/2010-04-01/Accounts/C3RTyd674gtalirS567q25687g57980gtr53/Messages/ABcDEFGhIJKLMNoPqRsTU1234565671z.json",
                               "subresource_uris": {"media": "/2010-04-01/Accounts/C3RTyd674gtalirS567q25687g57980gtr53/Messages/ABcDEFGhIJKLMNoPqRsTU1234565671z/Media.json"}}
2021-08-12 18:45:02,544 - 172.19.0.85 [12/Aug/2021:18:45:01 +0000] "POST /None?phone=%2B393312345678&message=The+Node+Exporter+instance+is+down,+please+check+soon+as+possible. HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0"

• Logs of the asterisk_asterisk_call

Some seconds or minutes after the SMS (configured in sms_before_call_wait_seconds) the asterisk_asterisk_call starts the calls round against the receiver number.

• Parameters from the config/caller_config.toml file

seconds_to_forget = 300
times_to_dial = 3

seconds_to_forget is the time window where asterisk_recall will try to recall the receiver. times_to_dial is the number of times to retry to call.

2021-08-12 18:47:02,655 - 172.19.0.81 [12/Aug/2021:18:47:02 +0000] "POST /asterisk_init?phone=00393312345678&message=The+Node+Exporter+instance+is+down,+please+check+soon+as+possible. HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0"
2021-08-12 18:47:22,690 Asterisk server 'http://192.168.122.234:8088' response: 201. Playing audio 'e23c1ebc.wav' to the channel '1628886822.1'
2021-08-12 18:47:22,694 Restoring the call control to the PBX on the channel '1628886822.1'
2021-08-12 18:47:22,694 - 172.19.0.81 [12/Aug/2021:18:47:22 +0000] "POST /play?asterisk_chan=1628886822.1&msg_chk_sum=e23c1ebc HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0"
2021-08-12 18:48:17,711 - 172.19.0.81 [12/Aug/2021:18:48:17 +0000] "POST /asterisk_init?phone=00393312345678&message=The+Node+Exporter+instance+is+down,+please+check+soon+as+possible. HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0"
2021-08-12 18:48:32,391 Asterisk server 'http://192.168.122.234:8088' response: 201. Playing audio 'e23c1ebc.wav' to the channel '1628886897.2'
2021-08-12 18:48:32,393 - 172.19.0.81 [12/Aug/2021:18:48:32 +0000] "POST /play?asterisk_chan=1628886897.2&msg_chk_sum=e23c1ebc HTTP/1.1" 200 178 "-" "Python/3.9 aiohttp/3.7.4.post0"
2021-08-12 18:48:32,393 Restoring the call control to the PBX on the channel '1628886897.2'

• Logs of the asterisk_recall

When the number 4 is not pressed by the receiver/callee, this component will recall again (times_to_dial).

2021-08-12 18:10:56,388 - Using the default path 'config/caller_config.toml'
2021-08-12 18:48:17,679 Retry to call phone number: '00393312345678' to play the message: 'The Node Exporter instance is down, please check soon as possible.' - Total retry period: '300' seconds

• Logs of the asterisk_ws_monitor

This component manages and records into the asterisk_ws_events table of the database the events of the Asterisk PBX when the py-phone-caller components are managing the call.

2021-08-12 18:47:22,695 Response for the playing audio 'e23c1ebc.wav' on the Asterisk channel '1628886822.1': '{"status": 201}'
2021-08-12 18:48:32,394 Response for the playing audio 'e23c1ebc.wav' on the Asterisk channel '1628886897.2': '{"status": 201}'

• Last but not least

The call_register component writes all the managed calls into the calls table. An example of table structure from the assets/DB/db-schema.sql file:

CREATE TABLE calls (
    id integer NOT NULL,
    phone character varying(64),
    message character varying(1024),
    asterisk_chan character varying(64),
    msg_chk_sum character varying(64),
    call_chk_sum character varying(64),
    unique_chk_sum character varying(64),
    times_to_dial smallint,
    dialed_times smallint,
    seconds_to_forget integer,
    first_dial timestamp without time zone,
    last_dial timestamp without time zone,
    heard_at timestamp without time zone,
    acknowledge_at timestamp without time zone,
    cycle_done boolean DEFAULT false
);

Manual testing

You can send messages and place calls from the terminal. This can be useful if you don’t intend to use the Prometheus monitoring system. You can trigger the messages or calls from cron or other scripts, for example.

An example of the payload sent from the Prometheus Alert Manager

The test-alert-from-the-alertmanager.json file:

{
   "receiver":"webhook",
   "status":"firing",
   "alerts":[
      {
         "status":"firing",
         "labels":{
            "alertname":"TestAlertFromAlertmanager",
            "instance":"localhost:9090",
            "job":"prometheus"
         },
         "annotations":{
            "description":"This text will be an audio message in order to verify if the setup is working",
            "summary":"our summary"
         },
         "startsAt":"2021-03-02T21:52:26.558311875+01:00",
         "endsAt":"0001-01-01T00:00:00Z",
         "generatorURL":"http://prometheus:9090/graph"
      }
   ],
   "groupLabels":{
      "alertname":"TestAlertFromAlertmanager",
      "job":"prometheus"
   },
   "commonLabels":{
      "alertname":"TestAlertFromAlertmanager",
      "instance":"localhost:9090",
      "job":"prometheus"
   },
   "commonAnnotations":{
      "description":"our description",
      "summary":"our summary"
   },
   "externalURL":"http://alertmanager:9093",
   "version":"4",
   "groupKey":"{}:{alertname=\"TestAlertFromAlertmanager\", job=\"prometheus\"}"
}

Make an HTTP POST request to simulate a Prometheus Alert.

$ curl --header "Content-Type: application/json" -X POST -d @ansible_py-phone-caller/alert.json http://127.0.0.1:8084/sms_before_call
{"status": "200"}

Start a Single Call from the shell

You can start a call from the shell using the curl command. Please pay attention to format of the number that’s using the 00 and the country code 39 because we’re using the Asterisk trunk without dial rules to modify the number. For example if you want to place a call in the UK, the first four numbers will be 0044.

Make a HTTP POST request to place a call from the terminal.

$ curl -X POST "http://127.0.0.1:8081/asterisk_init?phone=00392235896425&message=New%20message%20from%20the%20Fedora%20Server%20shell"
{"status": 200}

Send a Single SMS from the shell

Something similar to the previous curl command happens when you want to send a single SMS message. Twilio wants the + symbol instead of the 00 prefix. You will have to use %2B to encod the + symbol.

Make a HTTP POST request to send a SMS message from the terminal.

$ curl -X POST "http://127.0.0.1:8085/send_sms?phone=%2B392235896425&message=New%20message%20from%20the%20Fedora%20Server%20shell"
{"status": 200}

In future versions I hope to make this more user-friendly.

Finally, the classic ‘Wrapping Up’

By following the steps in this guide you can create a mini pager system. py-phone-caller isn’t intended to be used in mission critical environments because it is not yet ready for that kind of workload.

Thanks for reading. I hope this article was useful to you.

With more games being developed with the Godot engine, we need to learn how to package these games for Fedora.

Developing a game is complex. The requirements for each game differ. In the past developers created new game engines for each game. Over time game engines become more generic. They adapt to cover a style of game. Some engines can create a wide variety of games.

Godot is a well known open source game engine. Both open source and closed source games use the system. The Godot packages for Fedora run these games but no RPM package examples exist.

Much of the packaging is the same regardless of the application. RPM spec files need summary, version, license, description, etc. For build requirements, you need the godot-headless package. Godot publishes a pck file but requires a graphical user interface to run. Godot headless builds a project without needing a graphical user interface.

Linux cannot run a pck file. By adding godot-runner to the requirement section of the spec, the game can run on systems which install the RPM. The pck file is not completely platform independent But I believe it will work on all Linux systems. The build architecture can be set to noarch for this reason.

The same setup macro for another program can extract the tgz file. No macros exist for building Godot projects. You call godot-headless with ‐‐export-pack option. You specify the export name and the output filename. The install section simply needs to create a directory in $RPM_BUILD_ROOT/%{_datadir}/%{name} and copy the output file. Specify the files section and you are ready to build.

...

BuildRequires: godot-headless
Requires: godot-runner
BuildArch:      noarch

...

%build
godot-headless --export-pack Linux64 pigeonascent.pck

%install
mkdir -p $RPM_BUILD_ROOT/%{_datadir}/%{name}
install -p -m 644 pigeonascent.pck $RPM_BUILD_ROOT%{_datadir}/%{name}

%files
%{_datadir}/%{name}

...

Anyone can run the game by installing the package and running godot-runner with the specified pck file. Searching for the application on the GNOME desktop will not find it. Searching for the application in GNOME Software will not find it.

Beyond the minimum

Two files can fix these problems but you may need a third. The first file allows you to easily run the game from the GNOME desktop. The desktop file has the name of the program, how to execute the program, and other details.

[Desktop Entry]
Name=Pigeon Ascent
Comment=Take care of your own pigeon as they fight
Exec=godot-runner --main-pack /usr/share/pigeonascent/pigeonascent.pck
Terminal=false
Type=Application
Icon=pigeonascent
Categories=Game;RolePlaying;

In addition to the Exec entry, many desktop files include TryExec that tests if the executable exists in the path. If you include it, it would be godot-runner. In this example we call godot-runner. We could create a shell script to allow you to run the program easily from the command line.

For the icon you may be able to reuse the icon in the Godot project. If it is not the right size or you want a scalable vector version, you will need to create it. For Pigeon Ascent the icon image needs to be one pixel bigger in both directions.

The other file gives meta information to GNOME Software. It contains yet another name and description. It also has links to the website and screenshots. For games, Fedora includes the Open Age Rating System. This gives descriptors to allow parents to determine if the game content is acceptable. The Open Age Rating System generates this section after you enter the information for the game.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Copyright 2021 Dennis Payne <dulsi@identicalsoftware.com> -->
<component type="desktop-application">
  <id>pigeonascent</id>
  <metadata_license>FSFAP</metadata_license>
  <project_license>MIT</project_license>
  <name>Pigeon Ascent</name>
  <summary>Take care of your own pigeon as they fight</summary>

  <description>
    <p>
      Take care of your own pigeon as they fight increasingly stronger foes, and
      then facing the legendary Pigeon God at the end… can you keep death far from
      your bird?
    </p>
  </description>

  <launchable type="desktop-id">pigeonascent.desktop</launchable>

  <screenshots>
    <screenshot type="default">
      <image>https://static.jam.vg/raw/777/31/z/30974.png</image>
    </screenshot>
    <screenshot>
      <image>https://static.jam.vg/raw/777/31/z/3097c.png</image>
    </screenshot>
    <screenshot>
      <image>https://static.jam.vg/raw/777/31/z/30982.png</image>
    </screenshot>
  </screenshots>

  <url type="homepage">https://escada-games.itch.io/pigeon-ascent</url>

  <releases>
    <release version="1.5.2" date="2021-07-16" />
  </releases>

  <content_rating type="oars-1.1">
    <content_attribute id="violence-cartoon">moderate</content_attribute>
    <content_attribute id="language-humor">mild</content_attribute>
  </content_rating>
</component>

Some additional changes are needed to the RPM spec to include these files.

Adding gdnative

Sometimes Godot is insufficient to perform the tasks needed for a game. Gdnative allows C++ code to be added to a project. The gdnative code can be packaged as a separate project if it is used by multiple Godot projects. Unfortunately you need both godot cpp and header files which use the same download filename. Instead of using a url for the source tag in the spec file, specify the renamed file and include a comment above with the url.

# https://github.com/godotengine/godot-cpp/archive/refs/tags/godot-3.3.4-stable.tar.gz
Source1:        godot-cpp-godot-3.3.4-stable.tar.gz
# https://github.com/godotengine/godot-headers/archive/refs/tags/godot-3.3.4-stable.tar.gz
Source2:        godot-headers-godot-3.3.4-stable.tar.gz

Building gdnative assumes the godot cpp and headers are one directory above gdnative project. The following prep and build code builds the gdnative project.

%prep
%setup -q
cd ..
rm -rf godot-cpp
rm -rf godot-cpp-godot-3.3.4-stable
%setup -T -D -b 1
cd ..
rm -rf godot-headers-godot-3.3.4-stable
%setup -T -D -b 2

%build
cd ..
mv godot-cpp-godot-3.3.4-stable godot-cpp
rmdir godot-cpp/godot-headers
mv godot-headers-godot-3.3.4-stable godot-cpp/godot-headers
cd godot-cpp
scons platform=linux generate_bindings=yes -j4
cd ../%{name}-%{version}
mkdir bin
scons platform=linux

The install section copies the dynamic library to the library directory. If you build an executable for your godot project, the gdnative library works in the same directory. To avoid duplication we create the pck file and use godot-runner. When running with godot-runner, the library needs to be in the same directory as specified in the project. Typically this will be gdnative/linuxbsd.

Install the godot project the same as a project without gdnative. Add to the install section to create a gdnative/linuxbsd directory. Create a symlink to gdnative library. In order to run the game, you need to add a Path entry to the desktop file and set it to the directory with the pck file. This causes the program to run from that directory which allows it to find the gdnative library.

Where to go

Pigeon Ascent is awaiting approval to be added to Fedora along with Gdnativegamerzilla. My Pinball Disc Room requires Gdnativegamerzilla before submitting. The developers of Pigeon Ascent also created Diver Down which I recommend. Godot Wild Jams are good places to look for Godot games.

Development on Godot 4 continues. It will not be backward compatible. I do not know if Fedora will ship both versions or if we will need to convert games to the new Godot. They plan on having a tool to help with the conversion.