subtract or Add days, months and years to timestamp in Pyspark

In order to subtract or add days , months and years to timestamp in pyspark we will be using date_add() function  and add_months() function. add_months() Function with number of months as argument to add months to timestamp in pyspark. date_add() Function number of days as argument to add months to timestamp. add_months() Function with number of months as argument is also a roundabout method to add years to the timestamp or date. add_months()  or date_add() Function can also be used to add days, months and years to timestamp/date in pyspark. Let’s see an Example for each.

• Add days to timestamp/date in pyspark
• Subtract days to timestamp/date in pyspark
• Add months to timestamp/date in pyspark
• Subtract months to timestamp/date in pyspark
• Add years to timestamp/date in pyspark in roundabout way
• Subtract years to timestamp/date in pyspark in roundabout way

First lets create the dataframe as shown below

###### create dataframe in pyspark

import pyspark.sql.functions as F
from datetime import datetime
data = [
  ('George', datetime(2010, 3, 24, 3, 19, 58), 4),
  ('Andrew', datetime(2009, 12, 12, 17, 21, 30), 5),
  ('Micheal', datetime(2010, 11, 22, 13, 29, 40), 2),
  ('Maggie', datetime(2010, 2, 8, 3, 31, 23), 8),
  ('Ravi', datetime(2009, 1, 1, 4, 19, 47), 2),
  ('Xien', datetime(2010, 3, 2, 4, 33, 51), 3),
]

df = sqlContext.createDataFrame(data, ['name', 'birthdaytime', 'grad_Score'])
df.show(truncate=False)

We will be using the dataframe named df

Add days to timestamp/date in pyspark

To Add days to timestamp in pyspark we will be using date_add() function with column name and mentioning the number of days to be added as argument as shown below

### Add days to timestamp in pyspark

import pyspark.sql.functions as F 

df = df.withColumn('birthdaytime_new', F.date_add(df['birthdaytime'], 10))
df.show(truncate=False)

In our example to birthdaytime column we will be adding 10 days. So the resultant dataframe will be

Add months to timestamp/date in pyspark

To Add months to timestamp in pyspark we will be using add_months() function with column name and mentioning the number of months to be added as argument as shown below

### Add months to timestamp in pyspark

import pyspark.sql.functions as F

df = df.withColumn('birthdaytime_new', F.add_months(df['birthdaytime'], 3))
df.show(truncate=False)

In our example to birthdaytime column we will be adding 3 months. So the resultant dataframe will be

Add years to timestamp/date in pyspark

To Add years to timestamp in pyspark we will be using add_months() function with column name and mentioning the number of months to be added as argument as shown below, its a round about way in adding years to argument.

### Add years to timestamp in pyspark

import pyspark.sql.functions as F

df = df.withColumn('birthdaytime_new', F.add_months(df['birthdaytime'], 24))
df.show(truncate=False)

In our example to birthdaytime column we will be adding 2 years i.e 24 months . So the resultant dataframe will be

Subtract days from timestamp/date in pyspark

To subtract days from timestamp in pyspark we will be using date_sub() function with column name and mentioning the number of days to be subtracted as argument as shown below

### subtract days from timestamp in pyspark

import pyspark.sql.functions as F 

df = df.withColumn('birthdaytime_new', F.date_sub(df['birthdaytime'], 10))
df.show(truncate=False)

In our example to birthdaytime column we will be subtracting 10 days. So the resultant dataframe will be

Subtract months from timestamp/date in pyspark

To subtract months from timestamp in pyspark we will be using date_sub() function with column name and mentioning the number of days (round about way to subtract months) to be subtracted as argument as shown below

### Subtract months from timestamp in pyspark

import pyspark.sql.functions as F 

df = df.withColumn('birthdaytime_new', F.date_sub(df['birthdaytime'], 60))
df.show(truncate=False)

In our example to birthdaytime column we will be subtracting 60 days i.e. 2 months. So the resultant dataframe will be

Subtract year from timestamp/date in pyspark

To subtract year from timestamp/date in pyspark we will be using date_sub() function with column name and mentioning the number of days (round about way to subtract year) to be subtracted as argument as shown below

### Subtract year from timestamp in pyspark

import pyspark.sql.functions as F 

df = df.withColumn('birthdaytime_new', F.date_sub(df['birthdaytime'], 365))
df.show(truncate=False)

In our example to birthdaytime column we will be subtracting 365 days i.e. 1 year. So the resultant dataframe will be

for more details you can refer this

Other Related Topics:

• Get week number from date in Pyspark
• Get difference between two timestamps in hours, minutes & seconds in Pyspark
• Get difference between two dates in days, years months and quarters in pyspark
• Populate current date and current timestamp in pyspark
• Get day of month, day of year, day of week from date in pyspark
• Add Hours, minutes and seconds to timestamp in Pyspark
• Get Hours, minutes, seconds and milliseconds from timestamp in Pyspark
• Get Month, Year and Quarter from date in Pyspark
• Remove leading zero of column in pyspark
• Left and Right pad of column in pyspark –lpad() & rpad()
• Add Leading and Trailing space of column in pyspark – add space
• Remove Leading, Trailing and all space of column in pyspark – strip & trim space