How to efficiently combine date and time columns in Pandas with pd.to_datetime()?

Combining Date and Time Columns in Pandas with pd.to_datetime()

When working with date and time data in Pandas, it's often necessary to combine separate columns into a single datetime object. One way to do this is by using the pd.to_datetime() function.

Problem Statement

Consider a dataframe with 'Date' and 'Time' columns, as shown below:

Date      Time
01-06-2013  23:00:00
02-06-2013  01:00:00
02-06-2013  21:00:00
02-06-2013  22:00:00
02-06-2013  23:00:00
03-06-2013  01:00:00
03-06-2013  21:00:00
03-06-2013  22:00:00
03-06-2013  23:00:00
04-06-2013  01:00:00

Our goal is to combine these two columns into a single 'Date & Time' column.

Solution Using String Concatenation

One approach is to concatenate the 'Date' and 'Time' columns as strings and then convert the resulting string to a datetime object using pd.to_datetime():

result = pd.to_datetime(df['Date'] + ' ' + df['Time'])

This converts the concatenated string to a series of datetime objects.

Solution Using format Parameter

Alternatively, you can use the format parameter of pd.to_datetime() to specify the exact format of the combined string:

result = pd.to_datetime(df['Date'] + df['Time'], format='%m-%d-%Y %H:%M:%S')

This approach is faster than the previous one, especially when dealing with large datasets.

Performance Comparison

Using the %%timeit magic command, we can compare the performance of both approaches:

df = pd.concat([df for _ in range(1000000)]).reset_index(drop=True)

%timeit pd.to_datetime(df['Date'] + ' ' + df['Time'])

%timeit pd.to_datetime(df['Date'] + df['Time'], format='%m-%d-%Y %H:%M:%S')

The second approach with the format parameter is significantly faster for large datasets.

The above is the detailed content of How to efficiently combine date and time columns in Pandas with pd.to_datetime()?. For more information, please follow other related articles on the PHP Chinese website!