Logging & Configuration

Structured logging, log levels, and the 12-factor config pattern. The foundation of observable Python apps. FIND_VIDEO: search 'python logging configuration 12 factor' — recommended channel: ArjanCodes / mCoding. Aim for 10 min or under.

14 minutesVideo LessonPDF notes

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Motivation for Logging — Explains why structured logging replaces ad-hoc print statements to observe application trends and errors over time.
Standard Log Levels — Introduces the five standard log levels and demonstrates default suppression of DEBUG and INFO messages.
Configuring Threshold Levels — Demonstrates setting the root logger threshold to DEBUG using logging.basicConfig and numeric level constants.
Logging to Files — Configures the filename parameter in basicConfig to append execution records persistently to disk.
Customizing Log Format — Applies format string codes like asctime, levelname, and message to generate structured log outputs.
Applying Logging to Classes — Implements file-based INFO logging inside an Employee class to record instance creation events over time.

PDF notes

Bottom line: You will replace ad-hoc print statements with Python's built-in logging module to persist execution history and track system behavior over time. You will configure log severity thresholds, direct log output to persistent disk files, and customize log message formatting using standard LogRecord attributes.

This lesson on Logging & Configuration is hands-on and example-driven. You will replace ad-hoc print statements with Python's built-in logging module to persist execution history and track system behavior over time. You will configure log severity thresholds, direct log output to persistent disk files, and customize log message formatting using standard LogRecord attributes.

What You'll Be Able To Do

Import and configure Python's standard logging module without third-party dependencies.
Assign appropriate standard log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) to application events.
Redirect console log output to persistent disk files using logging.basicConfig.
Format log records using attributes such as timestamp, severity level name, and message content.

Detailed Concept Walkthrough

1. Standard Logging Severity Levels

Logging levels categorize diagnostic information into standardized severities so that messages can be filtered systematically rather than dumped unconditionally. Python defines five standard levels with numeric weights that determine whether an event meets the active capture threshold.

Threshold Evaluation: Python compares each log call's severity to the logger's threshold, emitting the record only if the call's level is greater than or equal to the configured minimum.
Under the Hood: Log level constants correspond to integer values in steps of 10: DEBUG is 10, INFO is 20, WARNING is 30, ERROR is 40, and CRITICAL is 50.
Default Behavior: The default root logging level is WARNING (30), which causes all DEBUG and INFO messages to be silently discarded unless explicitly reconfigured.
Level Semantics: DEBUG tracks granular diagnostics, INFO confirms expected operations, WARNING flags unexpected recoverable conditions, ERROR captures operational failures, and CRITICAL indicates application-halting crashes.

import logging

# Default threshold is WARNING; DEBUG and INFO do not output by default
logging.debug("Diagnostic detail for debugging")
logging.info("Confirmation of normal operation")
logging.warning("Warning: recoverable issue encountered")
logging.error("Error: operation failed")
logging.critical("Critical: application cannot continue")

Key Takeaway: Events below the active threshold are ignored entirely, making severity levels the primary tool for runtime log filtering.

2. Global Configuration with basicConfig

The basicConfig method provides a one-stop interface to set the global root logger's severity threshold, destination file, and output layout.

Mechanism: Calling logging.basicConfig with keyword arguments adjusts parameters like level, filename, and format before log records are emitted.
Syntax Rule: Level constants passed to the level parameter must be uppercase module attributes (e.g., logging.DEBUG), distinguishing them from logging methods like logging.debug().
Execution Flow: When configured, basicConfig sets up the root logger handlers; subsequent log calls across simple scripts inherit these global definitions.
Best Practice: Define logging.basicConfig once at the entry point of your script before executing any log-producing logic.

import logging

# Configure root logger with DEBUG threshold
logging.basicConfig(
    level=logging.DEBUG
)

# Now DEBUG messages will be captured and output
logging.debug("Add function executed with args: 10, 5")

Key Takeaway: logging.basicConfig applies global settings to the root logger and must use uppercase severity constants for the level argument.

3. File-Based Log Persistence

Writing log records to persistent storage files captures historical telemetry across multiple program runs instead of losing ephemeral console output.

Mechanism: Providing the filename argument to logging.basicConfig instructs the logging system to append log events to a physical disk file instead of standard stream output.
Under the Hood: Each execution appends new records to the designated log file without overwriting existing historical data from prior executions.
Best Practice: Use persistent log files in data pipelines and long-running services to monitor errors, analyze execution trends over time, and feed downstream visualization tools.

import logging

# Route all INFO and higher logs to a file
logging.basicConfig(
    filename="app_execution.log",
    level=logging.INFO
)

logging.info("Data extraction job started")
logging.info("Processed 500 records successfully")

Key Takeaway: Setting a filename in basicConfig automatically persists appended log records across script runs for long-term diagnostics.

4. Custom LogRecord Formatting

LogRecord attributes allow you to customize output strings into structured, human-readable entries containing execution context like timestamps and severity names.

Mechanism: The format argument in logging.basicConfig accepts formatted template strings that interpolate predefined LogRecord attributes using %(attribute)s syntax.
Key Attributes: Commonly used attributes include %(asctime)s for human-readable timestamps including milliseconds, %(levelname)s for the severity string, and %(message)s for the logged text.
Syntax Rule: Attribute placeholders must match Python's LogRecord attribute names exactly to prevent string interpolation errors during log dispatch.

import logging

# Configure custom layout: timestamp : severity : message
logging.basicConfig(
    filename="pipeline.log",
    level=logging.INFO,
    format="%(asctime)s:%(levelname)s:%(message)s"
)

logging.info("ETL batch completed successfully")

Key Takeaway: LogRecord format strings provide contextual metadata including timestamps and severity names alongside log payloads.

Topics Covered in Logging & Configuration

Motivation for Logging (0:00 - 1:35) — Explains why structured logging replaces ad-hoc print statements to observe application trends and errors over time.
Standard Log Levels (1:35 - 3:10) — Introduces the five standard log levels and demonstrates default suppression of DEBUG and INFO messages.
Configuring Threshold Levels (3:10 - 4:40) — Demonstrates setting the root logger threshold to DEBUG using logging.basicConfig and numeric level constants.
Logging to Files (4:40 - 5:55) — Configures the filename parameter in basicConfig to append execution records persistently to disk.
Customizing Log Format (5:55 - 7:35) — Applies format string codes like asctime, levelname, and message to generate structured log outputs.
Applying Logging to Classes (7:35 - 9:35) — Implements file-based INFO logging inside an Employee class to record instance creation events over time.

Python Cheat Sheet

import logging — Imports Python's standard logging module without extra dependencies
```
import logging
```
logging.basicConfig(level=...) — Sets minimum capture threshold using uppercase level constants
```
logging.basicConfig(level=logging.DEBUG)
```
logging.basicConfig(filename=...) — Directs output to a persistent disk log file
```
logging.basicConfig(filename="test.log", level=logging.INFO)
```
logging.basicConfig(format=...) — Defines log message structure using LogRecord attribute placeholders
```
logging.basicConfig(format="%(asctime)s:%(levelname)s:%(message)s")
```
logging.debug(msg) — Logs detailed diagnostic information (level 10)
```
logging.debug("Calculated total: %s", total_value)
```
logging.info(msg) — Logs standard confirmation of expected software behavior (level 20)
```
logging.info("Created employee instance: John Doe")
```
logging.warning(msg) — Logs unexpected events or non-fatal software issues (level 30)
```
logging.warning("Low disk space detected")
```
logging.error(msg) — Logs serious operational failures preventing specific function execution (level 40)
```
logging.error("Database connection timed out")
```

Comparison Table

Logging Level	Numeric Value	Intended Use Case
DEBUG	10	Detailed diagnostic problem investigation
INFO	20	Confirmation of expected normal behavior
WARNING	30	Unexpected events or non-fatal problems
ERROR	40	Serious failure executing a function
CRITICAL	50	Program-halting failure stopping entire execution

Common Pitfalls

Mistake: Expecting logging.debug or logging.info to print output under default settings. Avoid: Set level to logging.DEBUG or logging.INFO via logging.basicConfig because default threshold is WARNING.
Mistake: Passing lowercase function names like logging.debug as the basicConfig level argument. Avoid: Pass uppercase integer constants such as logging.DEBUG or logging.INFO to the level parameter.
Mistake: Relying on print statements for production error tracking and monitoring. Avoid: Use logging to persist time-stamped, severity-categorized records across multiple runs to persistent log files.

FAQs

Why does logging.basicConfig not display DEBUG logs in the console by default? Python's default logging threshold is set to WARNING, which automatically suppresses all DEBUG and INFO messages unless explicitly configured otherwise.
What is the difference between logging.DEBUG and logging.debug()? logging.DEBUG is an uppercase integer constant (value 10) used for configuration, whereas logging.debug() is the callable method that emits a log record.
Does writing to a log file overwrite existing logs upon re-running the script? No, logging.basicConfig appends new log records to the end of the specified file, preserving history across runs.
What are the common format codes for standardizing log message outputs? Common LogRecord attributes include %(asctime)s for timestamps, %(levelname)s for severity names, and %(message)s for the log text.

Common pitfalls

Mistake: Expecting logging.debug or logging.info to print output under default settings. Avoid: Set level to logging.DEBUG or logging.INFO via logging.basicConfig because default threshold is WARNING.
Mistake: Passing lowercase function names like logging.debug as the basicConfig level argument. Avoid: Pass uppercase integer constants such as logging.DEBUG or logging.INFO to the level parameter.
Mistake: Relying on print statements for production error tracking and monitoring. Avoid: Use logging to persist time-stamped, severity-categorized records across multiple runs to persistent log files.

Hey there, how's it going everybody? In this video, we'll be going over the basics of logging in Python. Now, there's a lot that we can do with logging in Python, so I'm actually going to break these up into two separate videos. Now in this video, we're going to look at how to get started with logging by replacing our print statements with log statements. We're also going to set different logging levels and also log information to files. Now, in the next video, we'll look at how we can use loggers throughout multiple modules and how we can configure the levels so that different information gets sent exactly where we want it to go. Um okay, so first of all, if you're not using logs right now, then you should definitely start trying to work it into your code wherever it would be useful. Uh Python comes with a logging module built in, uh so there's no need to install anything extra. A lot of people put off learning good logging habits because it's easier to sometimes just put in print statements and catch your problems on the fly. And there's nothing really wrong with that. I mean, print functions do have their place for sure. Uh but logging is definitely something that you'll need to have in place once your application grows beyond a basic project. Having good logs in place is going to allow you to, uh you know, look at behavior and errors over time and give us a better overall picture of exactly what's going on. And you can also pipe it into some visualization software to get a better perspective. Okay, so with that said, let's go ahead and get started. Uh so I have a file here with some very super simple functions and these aren't really useful for anything, but uh I wanted to start off with something easy so that we can focus on logging instead of what the code is doing. So I have four simple functions here: just add, subtract, multiply, and divide. And down here at the bottom, I'm using two number variables and these are set to 10 and 5 for now. And then I'm running through each of these functions and then I have some print statements here that are commented out. Okay, so if I just run this code as is, then it runs successfully with no errors, uh but we don't really know if it did everything correctly. So in order to print something out to the screen here, let's uncomment out all of these print statements. And let's go ahead and rerun that. So after we reran that code, we can see that the values that we printed out all showed up in our console down here and everything looks correct. Now, this is probably how a lot of us check our work as we go. But instead of using print statements, let's look at adding some very basic logging into this script. Now, first of all, we're going to have to import this. And logging is a built-in module, so all we have to do is just do import logging, and you don't need to install anything to get that to work. Okay, so before we log anything out, let's talk about logging levels. Now, logging levels allow us to specify exactly what we want to log by separating these into categories. Now, there are five standard logging levels and these are DEBUG, INFO, WARNING, ERROR, and CRITICAL. And I have these pulled up in my snippets here so that we can see a description of each of these. And I'm just going to copy and paste this into our file here. And I just grabbed this description uh directly from the Python documentation. So we can see here that DEBUG says that this is detailed information, typically of interest only when diagnosing problems. INFO is confirmation that things are working correctly or working as expected. WARNING is an indication that something unexpected happened or is indicative of some problem, uh maybe low disk space, and the software is still working as expected. Then we get down here to our errors, and due to a more serious problem, the software has not been able to perform a function. And then CRITICAL is a serious error, indicating that the program itself may be unable to continue running. So I'm just going to leave these commented here as we're walking through this video so that we can have that as a visual guide. Now, the default level for logging is set to WARNING, and what that means is that it will capture everything that is a warning or above. So by default, it will log WARNING, ERROR, and CRITICAL, and that means that it's going to ignore DEBUG and INFO log statements. So for example, down here at the bottom where we have these print statements, let's say that we wanted to turn these into DEBUG logging statements. So I could just say instead of print here, I'm just going to say logging.debug and save that. Now, the default behavior for these logging statements is just to log these to the console. So right now it's very similar to our print statement functionality. So now if I run this, then you can see that it doesn't log anything to our console, and that's because the default logging level is WARNING, and it's only going to log out warnings and higher. So it's not going to uh log these DEBUG and INFO statements. So instead, we need to change uh So first of all, let's just say that if we change these DEBUGs here to WARNING instead and run that, now you can see that we get some output to our console. Now, this has a little more information than our print function. So this tells us our logging level, which is set to WARNING. Um it also has this root here. Now, don't worry about that root right now and what that means. We're going to look at that in our next logging video. And now we have the message that we passed into our log here. Okay, so we've logged out some lines to the console, but we're not really using logging correctly right now because we have these set as warnings and they're not really warnings. We really wanted to log these lines as either DEBUG or INFO. Uh so we need to change our logging level to log those DEBUG and INFO statements out. So to do this, let's go up to the top of our file, and we're going to change some basic configurations here, and I'll just do this below our comments. And we can change these basic configurations with the basicConfig method. So I'm going to say logging.basicConfig. Now, this is camel case. And now to change that level, we're going to say level is equal to logging.DEBUG. Now, when you set that logging level, realize that this DEBUG here is in all caps. Now, that's different than our logging.debug method, um which we used earlier. This logging.DEBUG is all caps, and it's just a constant that's actually an integer in the background. And if I remember correctly, I think that all these constants are increments of 10. So I think DEBUG is actually just 10, INFO is 20, WARNING is 30, ERROR is 40, CRITICAL is 50, and so on. So now that we have our logging level set to DEBUG here, now let's change all of our warnings back to DEBUGs. So I'm going to go down here where we did logging.warning, and instead I'm going to say logging.debug. And now let's go ahead and rerun this. Okay, so now we can see that it's printing our DEBUG statements there in the console. Now, so this is good so far, but you may have come to this video expecting to see actual log files and not just printing information to the console. And log files are a great way to capture information because it allows us to see our logged information over time and in one place. So instead of logging this information to the console, let's instead create a log file. Now, to do this, it's just as easy as specifying a file in our basicConfig method. So the keyword that we're going to use here is filename and set that equal to whatever we want our log file to be. And I'm just going to call this test.log for now, and we also need a comma there to separate those keywords. Okay, so now let me rerun this again. Now, nothing showed up down here in our console. Now, I'm not sure if you can see my file system very well up here, but this new file appeared: test.log. Now, I'm going to go ahead and open this up, and you can see that it logged all of the information from our script. So now if I go back to my script and I change these numbers—so instead of 10 and 5, I'm just going to use 20 and 10—and now if I rerun this and then open back up this test.log, now you can see that it has our previous values that ran and the uh the latest run here. So now we're going to be able to see our logged information over time and over multiple runs. Okay, so one more thing that I want to show you in this basic logging video is how to change the format. And what I mean by the format is if we look back at our log file here, you can see that it has the log level and the logger, which we said that we weren't going to cover in this video, we'll cover that in the next video, and then it has the message here. Um now, to change the format in our log, we need to add in some special values to our basicConfig. Now, to view what is available to us, I have the options pulled up here over here in my browser. Now, these special format codes right here are what uh is going to allow us to change our format, and you can see a description of each of these. So we have the uh human-readable time, we have the file name, the level name, uh the line numbers, the messages, and things like that. So let's say that we wanted to change our logging format to be the time and the level name, and then the message. So I'm just going to go ahead and copy each of these. So first, I'll go ahead and grab the time here and copy that. And now to set this formatting, I'm going to go back to my sample file, back up here to the top, and in my basic configuration, I'm just going to add in another keyword, and that next keyword is format, and I'm going to set this equal to I'm just going to paste this in what we copied over for the time. And then I'm going to separate that with a colon there, and now I'll go back over and grab the level name, so we want the level name format. I'll copy that and paste that in, separate that with a colon. And now we also want the message, so I'll grab the message and go over and paste that in. So now if I save this and I run that, and now I go back to my log file, now you can see that the format down here with our latest run, the format of our logs has changed, and now we have the time—and this comma here is just uh dealing with millisecond data—and then we have the log level, which is DEBUG, and then we have our message that we printed out. Okay, so now let's take a look at another module and add some logging to it real quick just so that we can walk over all this one more time a little bit more quickly. So the module that I have here is one that I pulled from my object-oriented tutorial series. Now, if you don't know how classes work or exactly what this code is doing, then don't worry about it too much. Uh we're just going to use this as an example to add some logging. So just a quick overview of what's going on, I have this really small Employee class here, and when we create an employee, it comes in here and sets the first name and the last name uh on the instance. And then we're printing out that we created an employee. And then down here at the bottom, we're actually creating two instances of two new employees. So with this print statement in here, it should tell us when these two instances are created. So I'm going to go ahead and run this code, and you can see that it tells us that our two instances were created, and we have our print message there. Okay, so now let's go ahead and add some logging. Let's say that we wanted to see, you know, what employees were created over time. Uh so we're going to go through the same process that we did in the other uh module, except we're just going to go a little faster here. So we'll go ahead and import logging, and then we'll set up our basic configuration. I could copy this over from the other file, uh we'll just go ahead and type this in real quick. So I'll say logging.basicConfig, and then within here, we'll do a filename, and we'll set that filename equal to employee.log. Now, that's the log file that we're going to log to. Now, for our level here, remember that the default is set to WARNING. Um let's say um let's set this equal to INFO. I think that the uh our employee creation would be a good INFO log level. And lastly, now let's go ahead and set the format for uh our logs. So to do this, let's do format equals and within this string Now, this one I am going to grab from the other example since these uh formatting codes can be a little confusing. Now, I'm just going to go ahead and grab the level name and the message here, so I'll copy that and just paste that in there. Now, remember if you want to see everything that is available to you here, then they do have this in the Python documentation. It is up here in the Python 3 logging library documentation down here at LogRecord attributes. Okay, so now that we have our logging configured here, now let's replace our print statement with a log INFO call. So instead of print, I'm just going to say logging.info and pass in the exact same message. And just to make sure that this is working with new information, let's go ahead and create one more employee down here at the bottom. I'll call this employee_3, and we'll just call this call her Jane Doe, save that. Okay, so now if I run this code, then nothing popped up in our console, but over here in the file system, we have this employee.log. I'm going to open that up, and here you can see that we have all of our logged information uh at the INFO level and the custom formatting that we added in. Okay, so I think that is going to do it for this video. Uh we've gone over how to use basic logging instead of print statements. We also learned about the different log levels and how to change those levels, um also how to log to a file and also how to format those log messages. So for small applications, logging like this will be a good start, but there are some issues that we can run into once we start importing our other modules uh because they all try to share the same logger. Um so in the next video, we'll learn how to create separate loggers, um add handlers and formatters to those loggers, and also how we can log our information to multiple locations. And I'll leave a link to that next video in the description section below. Now, if anyone has any questions about what we covered in this video, then feel free to ask in the comment section below and I'll do my best to answer those. Now, if you enjoy these tutorials and would like to support them, then there are several ways you can do that. The easiest way is to simply like the video and give it a thumbs up, and also it's a huge help to uh share these videos with anyone who you think would find them useful. And also if you have the means, you can contribute through Patreon, and there's a link to that page in the description section below. Be sure to subscribe for future videos, and thank you all for watching.

Frequently asked questions

Why does logging.basicConfig not display DEBUG logs in the console by default?

Python's default logging threshold is set to WARNING, which automatically suppresses all DEBUG and INFO messages unless explicitly configured otherwise.

What is the difference between logging.DEBUG and logging.debug()?

logging.DEBUG is an uppercase integer constant (value 10) used for configuration, whereas logging.debug() is the callable method that emits a log record.

Does writing to a log file overwrite existing logs upon re-running the script?

No, logging.basicConfig appends new log records to the end of the specified file, preserving history across runs.

What are the common format codes for standardizing log message outputs?

Common LogRecord attributes include %(asctime)s for timestamps, %(levelname)s for severity names, and %(message)s for the log text.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.