Looks like the issue is at
insertInto Try using the
saveAsTable instead, insertInto is deprecated in spark 1.4.
dataToInsert.write.mode("append").saveAsTable("SchemaName.animals") note that schema name is mandatroy.
if the above one leaves you with no luck try writing to the hive tables HDFS path. I would also suggest you repair the hive table after running the pyspark job sql command for it :
msck repair table <schema.tablename>; either from hiveContext or hive.
Why do you need msck repair ?? check out my other answer here
Here is Doc snapshot :
saveAsTable(name, format=None, mode=None, partitionBy=None, **options) Saves the content of the DataFrame as the specified table.
In the case the table already exists, behavior of this function depends on the save mode, specified by the mode function (default to throwing an exception). When mode is Overwrite, the schema of the DataFrame does not need to be the same as that of the existing table.
append: Append contents of this DataFrame to existing data.
overwrite: Overwrite existing data. error: Throw an exception if data already exists.
ignore: Silently ignore this operation if data already exists. Parameters: name – the table name format – the format used to save
mode – one of append, overwrite, error, ignore (default: error)
partitionBy – names of partitioning columns options – all other string options New in version 1.4.