How to Debug Ansible Collections using Pycharm

Background

When developing new Ansible modules or trying to understand an existing module better, it can be helpful to take advantage of all the features of Pycharm to debug the Python code in the module.

To use the debug features in Pycharm the Python code must be run from within Pycharm. However, the Python code in Ansible is called as a result of running a playbook. Therefore we need a way to run our playbook as we would normally but somehow trigger a Pycharm debug session at the desired point in the Python code. This post describes a method to do this.

Prerequisites

  • Local and remote machines – local running Pycharm, remote running the Ansible/Python code. Remote can be a separate physical machine or a VM on your local.
  • Must be Pycharm Professional Edition on your local machine.
  • SSH connectivity between local and remote machines.

Setup

  1. On your local machine, set up a new, empty Pycharm project.
  2. Copy the Python file(s) that you want to debug, from the remote server to the new Pycharm project on your local machine.
    • Note: It is likely that in the course of debugging the file, you will want to use the Pycharm ‘Step Into’ feature to debug other functions/modules that the file uses. For ‘Step Into’ to function correctly, the files containing these other functions/modules must also be copied from remote to local. Therefore, unless you are sure that you only want to debug a single file, it is probably easiest to copy the entire Python structure (containing the files you want to debug) from the remote to the new Pycharm project on your local machine
  3. Configure a new Run/Debug Configuration
    • Use the + to set up a new Python Debug Server
    • Give the new server a name
    • Specify the IP of your local machine
    • Pick any unused port (12345 will usually work)
    • Add a Path Mapping.
      • Click the grey folder icon at the end of the Path Mappings field.
      • Click the ‘+’ to add a new mapping.
      • Add a local and remote path. The local path should be the root folder of the new Pycharm project on your local machine and the remote path should be the root folder of the project on the remote machine.
    • Select ‘Redirect output to console’ and ‘Suspend after connect’
    • Copy the 3 commands that are specified in the ‘Update your script:’ section of the setup window to a text file on your machine for use later
    • Select ‘OK’
  1. On the remote machine run the ‘pip install pydev-pycharm’ command you copied earlier
  2. On the remote machine open the python file that you want to debug.
  3. Scroll to the point in the file where you want the breakpoint and paste in the ‘import pydevd_pycharm……..’ and ‘pydevd_pycharm.settrace……..’ commands you copied earlier.
  4. Save and close the file.

Start the Debug

  1. On your local machine, run the new Python Debug Server by hitting the green bug icon
  2. The debug pane should open with a ‘Waiting for process connection…’ message at the bottom of the screen.
A close-up of a computer screen

Description automatically generated
  1. On the remote machine, run the playbook that uses the Python file you want to debug.
  2. When the code reaches the point where you have inserted the breakpoint, the remote machine will connect to your local machine and the debug window will open.
A screenshot of a computer

Description automatically generated
  1. Click on ‘Debugger’ to see the stack and variables:
  1. Debug the code as you would normally using the ‘Step Over’, ‘Step Into’ buttons, etc.

Quick Method

There is a quicker way to start a debug but it does come with significant shortcomings.

It is possible to omit ‘Setup – step 2’ and ‘Setup – step 3 – Add a Path Mapping’.
If you omit these steps, when you start the debug, you should be presented with a window similar to below, with the ‘Edit settings, Auto-detect, and Download’ options.

If you select download, the file you want to debug will be downloaded and the debug session will begin:

However, the cursor that normally highlights the position of the debug in the code, is not present. The position of the debug can be determined by looking in the yellow ‘thread’ pane and looking at the line number of the topmost thread (in the above example, 62).
Additionally, although ‘Step over’ works, ‘Step into’ will not work.

If you just want to do a quick check of something like the value of a variable, then this quick method may be enough for you.

Sending Messages from Web Server to Web Client using Websockets, Redis and Webdis

Sometimes your web application will initiate some server-side code that runs for an extended period. This can be a problem for a user left staring at a web page with no feedback – the temptation is to re-submit, go back or even close the application. I found this to be particularly relevant while working on network automation applications that involve a lot of relatively slow interactions with network devices.
What is needed is a way to continuously update the user with messages so that they can see that the application is functioning correctly.
This post will describe how to use WebSockets, Redis, and Webdis to send messages from the server-side of your web application to the client.

The web client “subscribes” to the message queue and the server “publishes” to the queue. Every message published by the server is seen by the client. The client does not have to request the message.
Continue reading Sending Messages from Web Server to Web Client using Websockets, Redis and Webdis

How do Python imports and __init__.py files work in a package

Python packages typically utilize mechanisms like imports, __init__.py files, and inheritance to create scaleable, and reusable code. In this post, we will look at how these mechanisms work by exploring a piece of code that uses the Netmiko library.

Continue reading How do Python imports and __init__.py files work in a package

Using Parameter Substitution for Column Names with Python MySQL Connector

The use of data parameter substitution with MySQL Connector is well documented:

To create a new row in a table by passing in multiple data parameters the syntax might be:

cur.execute('INSERT INTO table1 (name, address, phone, postcode) values (%s, %s, %s, %s)', ('John', '1 Front Street', '416-111-1234', 'N4R2E5'))
db.commit()

Where the %s are placeholders for the data values being passed in the following list.

Continue reading Using Parameter Substitution for Column Names with Python MySQL Connector

Viewing Ingress CoS Markings on a Juniper JUNOS MX Series Router

Introduction

When troubleshooting CoS related problems or testing CoS features it is often useful to be able to view the class of service of packets as they enter your router. If packets are not entering your router with the correct markings they will not be assigned to the correct forwarding class and will not receive the desired CoS treatment. Oddly, there does not seem to be a JUNOS command that displays counters of ingress markings. One way around this is to use an inbound firewall filter that matches and counts ingress packets based on their markings.

Continue reading Viewing Ingress CoS Markings on a Juniper JUNOS MX Series Router

Learning Python 5 – Practical Projects

As with learning anything, practice is the key and with Python that means typing code into the CLI – you will not make much progress just reading blogs or watching tutorials. In an effort to learn quickly it can sometimes be tempting to think, “I will watch the next video, then I will try a practical exercise”. It’s not until you try and code something that you realize you have not taken it in as well as you thought.

Continue reading Learning Python 5 – Practical Projects

Learning Python 3 – Tutorials

There are so many Python tutorials available online that sometimes it is difficult to know which one to use. The quality can vary significantly and sometimes the author may be too fast or too slow or just have a style that does not work for you. The following authors are my favorites based on their speed of delivery, style of delivery, use of examples, and overall consistent quality.

Continue reading Learning Python 3 – Tutorials

Learning Python 2 – Online Training

Which online training sites have good content?

Paid online training can provide a more structured approach to learning. I have found it most useful to start with some paid online training courses and then supplement or fill in the blanks with free online tutorials. I have found the following sites useful:

Continue reading Learning Python 2 – Online Training