Surveys conducted as part of pre- and post-occupancy evaluations (POEs) are a valuable data collection method for continuous process improvement. These systematic evaluations help foster a high level of engagement with clients as the data from the surveys provide valuable insights into the attitudes, motivations, and pain points of the building occupants. These insights, moreover, prove instrumental in informing future facility design.
Analyzing qualitative data from surveys traditionally involves a time-consuming process of manually reviewing large volumes of text-based data, identifying recurring themes and sentiments, and finally extracting key insights.
However, the relatively recent introduction of artificial intelligence (AI) is drastically revolutionizing the process of analyzing and interpreting this data. A task that would typically take weeks to complete manually can now be completed in days. As AI tools continue to evolve, it will be important for organizations to leverage them in order to continue making efficient data-driven decisions and maintain a competitive edge.
As part of design research, Flad's Innovation and Analytics team tested a few AI tools to analyze over 2,500 open-ended responses from a previously conducted master planning survey and compared it to the manually analyzed findings of the same dataset.
One of the main benefits of manually analyzing data is a more nuanced understanding based on lived experiences and human emotions. However, reading and thematically tagging each response, especially in large datasets, proves to be not only time-consuming but also prone to human error due to subjectivity and misinterpretation.
Utilizing AI in Qualitative Research
Surveys conducted as part of pre- and post-occupancy evaluations (POEs) are a valuable data collection method for continuous process improvement. These systematic evaluations help foster a high level of engagement with clients as the data from the surveys provide valuable insights into the attitudes, motivations, and pain points of the building occupants. These insights, moreover, prove instrumental in informing future facility design.
Analyzing qualitative data from surveys traditionally involves a time-consuming process of manually reviewing large volumes of text-based data, identifying recurring themes and sentiments, and finally extracting key insights.
However, the relatively recent introduction of artificial intelligence (AI) is drastically revolutionizing the process of analyzing and interpreting this data. A task that would typically take weeks to complete manually can now be completed in days. As AI tools continue to evolve, it will be important for organizations to leverage them in order to continue making efficient data-driven decisions and maintain a competitive edge.
As part of design research, Flad's Innovation and Analytics team tested a few AI tools to analyze over 2,500 open-ended responses from a previously conducted master planning survey and compared it to the manually analyzed findings of the same dataset.
One of the main benefits of manually analyzing data is a more nuanced understanding based on lived experiences and human emotions. However, reading and thematically tagging each response, especially in large datasets, proves to be not only time-consuming but also prone to human error due to subjectivity and misinterpretation.
We tested a few large language models (LLM) and a software program primarily built for qualitative data analysis. LLMs are a type of machine learning model designed to understand large amounts of text data with the capacity to be trained to generate summarized human-like content.
Many of these models can be prompt-engineered to enable predictive capabilities but currently are also prone to inherit biases and inaccuracies based on the data they are trained on.
On the other hand, the qualitative data analysis software uses structured programming languages and coding for in-depth qualitative analysis and has recently incorporated AI capabilities, helping to automate the analysis process.
The findings were informative and compelling. Although the idea behind sorting and organizing data remains the same across the tools, they differ in their approach.
While all tools were able to process the information and generate effective outputs within a matter of minutes, each of them left something to be desired. The qualitative data analysis software program performs well at auto-generating codes with a capability to track coded themes, which is very useful in the analysis process. But this sophisticated program, with its advanced functionalities, comes with a steep learning curve and can feel futile for small-scale survey analysis.
Conversely, the LLMs, although easier to use, currently lack the desired consistency and reliability in their outputs, which makes accurate reproducibility unattainable despite data training. This could be attributed to the fact that the current models do not have subject matter expertise in the field of architecture.
There continues to be a palpable sense of momentum regarding the potential of AI across diverse domains. However, it remains vital for human oversight to be an inherent aspect of AI integration.
As published in an article by Cornerstone Editors in 2024, AI systems operate based on patterns and algorithms and struggle to adapt to dynamic situations or understand the subtle nuances of human interactions and sentiments, which are commonly found in survey responses.
Moreover, the article states that while AI algorithms can be highly efficient, they cannot assess and prioritize ethical considerations. Security concerns about personal data protection while using AI, although justified, are risks that have been faced through decades of internet and smartphone usage. Therefore, continuing to integrate internal governance mechanisms such as data anonymization while using AI will be vital in protecting confidential client information.
Granted that this testing provides a static snapshot of existing AI capabilities, the continued application of AI in qualitative research remains inevitable. Currently, these tools are most effective at summarizing open-ended survey responses into valuable and concise summaries alleviating the overall time taken in extracting insights.
Considering the evolutionary nature of new technology, Flad’s Innovation and Analytics team will continue to explore and test AI tools with the goal of firm-wide knowledge sharing to help improve overall efficiency and workflow.