I figured that the error was caused due to strict pydantic definitions, which stop us from defining any new instance variables in the inherited class definition. Although I could not find any workarounds that worked with llama_index, langchain had more flexible definition so switched packages in the end.