Using data of a CSV

Hi all,

Does somebody have an example of using a csv table as you would using a pandas df or dict?

I only found the following example below, but this is just to read one value of the cvs. Thanks!

class Controller(ViktorController):
…

@ParamsFromFile()
def process_file(self, file: File, **kwargs):
    ...

    return {
        'tab': {
            'section' : {
                'number_of_entries': number_of_entries,
                'project_name': project_name
            }
        }
    }

Hi Bart,

As you suggested, you can parse a csv using the Pandas library. The csv table is then converted into a DataFrame object from which you can extract any information that you can to store into your parametrization. Keep in mind that the parametrization of your entity can only include serializable objects, so no DataFrame but you can always convert your DataFrame into a dictionary.


from io import StringIO
import pandas as pd

@ParamsFromFile(file_types=[".csv"])
def process_file(self, file: File, entity_id: int, **kwargs) -> dict:
        file_content = file.getvalue("utf-8")
        df = pd.read_csv(StringIO(file_content))  
        serialized_df = dataframe.to_dict()


    return {}




That easy! Thanks a lot. It saves me a lot of trails and errors :slight_smile:

1 Like

Hi Matthias,

Unfortunately I’m not able to use the dict in other functions of the controller (for example use data from the dict in a Plotly graph). I’ve tried Memoize, but I couldn’t get it working with the ParamsFromFile().

Any ideas?

Hi Bart,

The decorator @ParamsFromFile should allow you to save a dictionary into the parametrization of your entity. You can use a HiddenField (here) to store objects in params without having an UI Field connected on the editor.

You can then retrieve this dictionnary in any method of your controller with the params argument


class Parametrization(Parametrization):
    section.my_dict = HiddenField("UI name", name='my_dict')

class Controller(ViktorController):
    @ParamsFromFile(file_types=['.csv'])
    def process_file(self, file: File, **kwargs):
        # Insert project specific content: convert csv into serialized dict
        my_dict = ...
        return {'my_dict': my_dict} # The structure of this dictionnary should match your parametrization.py file

    def controller_method(self, params, entity_id, **kwargs):
        my_dict = params.my_dict # type is Munch at this point




Hi Matthijs,

Thanks for your clear answer. I couldn’t have thought of this myself. Thanks!

Bart

Hi @mhauth!

Im trying to do something similar as BartG, trying to create a upload field for .CSV files.
My purpose for this is uploading a electric demand profile, which i want to visualize and do calculations with afterwards(preferably in a pd.DF).

I tried modifying/copying the proposed script in this thread but i gives me the following error: "
ERROR : App definition is invalid: Entity type Controller is wrongly defined: File entity types are not supported in editor app type, because entity creation is not possible."

I suppose the error means a “tree” app-type is neccesary? I am new to Viktor and was just getting to understand the editor-type. So wanted to check if this is really neccesary or if there are other options.

Kind regards,

#parametrization:
    step_a.file = FileField(
        "Upload-usage",
        file_types=[".csv"],
        max_size=10_000_000,
    )
    step_a.e_profile = HiddenField(
        "E-profile",
        name='my_dict'
    )
#Controller:
    @ParamsFromFile(file_types=['.csv'])
    def process_file(self, file: File, **kwargs):
        # Insert project specific content: convert csv into serialized dict
        file_content = file.getvalue("utf-8")
        string = StringIO(file_content)
        my_dict = pd.read_csv(string)
        my_dict = my_dict.to_dict()
        return {'my_dict': my_dict}  # The structure of this dictionnary should match your parametrization.py file

    @PlotlyAndDataView("Data", duration_guess=10)
    def get_usage_data(self, params: Munch, **kwargs):
        energy_df = Controller.process_file(params.step_a.file)
        fig = dict(data=[
            {"type": "bar", "x": energy_df.index, "y": energy_df.iloc[:, 1], "name": "Energy Demand"},
        ], layout={
            "title": {"text": "Electricity Demand over time."},
            "xaxis": {"title": {"text": "Date"}},
            "yaxis": {"title": {"text": "Demand [kWh]"}},
        })
        data = energy_df
        return PlotlyAndDataResult(fig, data)

Hello Tigerfire, welcome to the forum!
The post you are referring to is quite old. In the meantime, there have been some nice developments, such as the FileField. With the FileField you can upload files easily, so you can do all this within the editor type. Below is an example of how to open a csv file from the filefield:

Class ExampleParametrization(Parametrization):
    filefield_example = FileField('CSV upload')

class Controller(ViktorController):
    label = 'My CSV app'
    parametrization = ExampleParametrization

    def do_something_with_csv(self, params, **kwargs):
        upload_file = params.filefield_example.file
        data_file = BytesIO(upload_file.getvalue_binary())
        df = pd.read_csv(data_file)

Let me know if this help you solve your problem!

Hi Jelle,

Thank you for your quick response.
After resolving some other errors, related to formatting the right objects and method calls.
I managed to at least print the dataframe! So thanks alot!

I do get the error: "ERROR : Result can not be serialized to JSON: Object of type DataFrame is not JSON serializable "

But i guess this is the same problem discussed earlier in this thread? Or would it be possible to call the DF in a @DataView method directly from params aswell?

Kind Regards,

I’m not sure what you’re trying to do, could you provide more context to the error message you get?

If you want to get the dataframe from a fixed path, you can use:

df = pd.read_csv(Path(__file__).parent / "sample.csv")

Just make sure you put the csv file in the app folder.
For more information on the dataview, please check out our documentation

So on the left side (parametrization) i want a CSV upload field, and if a file gets uploaded and selected i want to visualize the data on the right side.

So current situation:

Parametrization file:
    step_a = Step("Upload your Usage",
                  views=['visualize_data'],
    )
    step_a.text = Text(
        """# Please Upload your electricity demand profile!
In case you are not able to do this select your building type to generate a synthetic profile.
"""
    )
    step_a.upload = FileField(
        "Upload-usage",
        file_types=[".csv"],
        max_size=10_000_000,

Controllerfile:
    def do_something_with_csv(self, params, **kwargs):
        upload_file = params.step_a.upload.file
        data_file = BytesIO(upload_file.getvalue_binary())
        df = pd.read_csv(data_file)
        df = df.to_dict()
        return df

    @DataView("OUTPUT", duration_guess=1)
    def visualize_data(self, params, **kwargs):
        eprof_df = Controller.do_something_with_csv(self, params)
        eprof_df = pd.DataFrame(eprof_df)
        print(eprof_df)
        value_date = eprof_df['Time.1']
        print(value_date)
        value_energy = eprof_df['PV Productie (W)']
        data = DataGroup(
            DataItem('Data item 1', value_date),
            DataItem('Data item 2', value_energy)
        )
        return DataResult(data)

TypeError: Object of type Series is not JSON serializable
But at this point it is more a vizualisation error, ill find it out with documentation.
Thanks for your help