ChatGPT and the Generation of Digitally Born “Knowledge”: How Does a Generative AI Language Model Interpret Cultural Heritage Values?

1. Introduction

At the time of writing, artificial intelligence (AI) has reached public consciousness, with a wide-ranging debate on its present and potential future abilities, its dangers and the ethics of its usage. All of this was brought about by the public release of DALL-E (an image generator) and ChatGPT in early 2022. The Chat Generative Pre-trained Transformer (ChatGPT), a generative AI language model developed by OpenAI, is a type of deep learning model that uses transformer architecture to generate coherent and contextually relevant, human-like responses based on the input it receives [1].

From its formal release in 2018, ChatGPT has undergone several iterations and improvements. In November 2022 ChatGPT 3.5 was released to the general public, as a part of a free research preview to encourage experimentation. The current version GPT-4, released in March 2023, was trained (by human trainers) on data set of 175 billion parameters and reputedly exhibits greater factual accuracy, reduced probability of generating offensive or dangerous output and greater responsiveness to user intentions as expressed in the questions/ query tasks [2]. Analysis has shown that the ChatGPT language model memorized a wide collection of books (fiction, non-fiction and scientific), with the degree of memorization correlated with frequency with which passages of those text appear on the web [3,4]. The temporal cut off for the addition of training data was September 2021, which implies that ChatGPT cannot integrate or comment on events, discoveries and viewpoints that are later than that date. It is asserted in public media, however, that GPT-4 has the ability to search the Internet in real time.

ChatGPT has been shown to be capable of writing lines of code [5], producing poetry [6], short stories and plays [7,8,9], English essays [10], as well as producing simulated scientific content (see below)

1.1. The use of ChatGPT in academic disciplinary research

There is a growing body of literature that examines the level of knowledge of ChatGPT as reflected in its responses to several fields of research, such as chemistry [11], the use of remote sensing in archaeology [12], architecture [13], diabetes education [14], medicine [15,16,17,18,19], nursing education [20], agriculture [21] and computer programming [22]. Some papers looked at the use of ChatGPT to write literature reviews [23,24,25].

In addition, some work looked at the ‘perceptions’ of ChatGPT about its future role in some disciplines and professions, such as accounting[26], libraries [27], academia in general [27], medicine [15], medical research [28], health care [29] digital leadership and technology integration [30], or textile manufacturing [31]. Several of these papers make use of the interactive nature of ChatGPT and frame their research as a conversation about the topic [11,12,15,32]. Other research has examined the role and usefulness of CHatGPT in advising or guiding professionals, for example in the fields of medicine such as arthroplasty [33,34], nursing [6], dentistry [35], orthopedic [36] and pediatric research [28].

A growing body of research has been examining the effects of ChatGPT on education and academia in general. At the time of writing, there are two discrete strands of thinking: one that considers ChatGPT as a potential device to enhance learning [35,37,38,39,40,41,42,43,44]; and one that considers its effect on assignment writing and associated student misconduct [32,40,41,45,46,47,48,49]; as well as the integrity of academic writing and publishing in general [50,51,52,53,54,55,56,57,58,59].

As several authors noted, ChatGPT is the typical double-edged sword presented by many new technologies: both useful and detrimental [60]. In response to the threat of AI-generated text to the integrity of assignments and other text, tools have been developed (and are being refined) to assess a block of text for it human vs AI authorship [61,62]. On the other, techniques to evade detection are also being examined [63,64].

1.2. ChatGPT in cultural heritage research

Compared to disciplines such as medicine, there has been little formal exploration of the use of ChatGPT in cultural heritage research and management. So far it has been explored in terms of its knowledge of the potential for remote sensing in archaeology [12], and creating plain language summaries of archeological research reports [65]. Other work looked at the ethical implications of generating ‘new’ content, in textual and pictorial form, and the limitations ChatGPT (and its image generation cousin DALL-E) possess in the cultural sphere [66,67].

The majority of the current discussion on the use of ChatGPT in cultural heritage contexts occurs in the unrefereed blog sphere, with the majority of activity in the museums and collections sphere.

The potential of ChatGPT as a tool for message generation, content marketing and audience interaction can be readily ported to a museum context [68]. There is an emerging potential to use sentiment analysis of general visitor enquiries, specific queries and overall feedback to provide an integrated understanding of visitor interest and reactions to specific exhibitions or the museum overall and to track visitor satisfaction [68,69].

It has been posited that that ChatGPT (or future iterations) can be readily used as generic document and content analysis to extract pertinent data from longer documents and to provide a succinct summary [70], which could be employed in the creation of exhibition texts [71,72] and scripts for audio guides [72], while data extraction from object inventory data can result in the creation of exhibit labels and catalogue information [72] and museum guides [73]. This can be extended in the form of customized tours of exhibits and museum holdings that reflect a visitor’s personal interests or information needs [68,69]. This is readily implementable given the increased digitization of museum holdings and their presentation to the public in the guise of digital exhibits, which was supercharged by the digital pivot required in response to the shuttering of museums during the COVID-19 pandemic [74,75,76,77,78]. Not surprisingly, some concerns have been raised as to how ChatGPT could affect museum studies projects in school curricula encouraging plagiarism and reducing active enquiry [79].

While ChatGPT appears to have considerable success in responding to fact-based tasks [37,47], the question arises whether it can provide similarly valid responses when tasked with explaining theoretical concepts.

1.3. The nature and assessment of values attributed to cultural heritage assets

In broad terms, cultural heritage is the result of peoples’ interactions, both with each other and with the environment in which they live. The outcome of these interactions expresses itself in a number of forms. Peoples’ interactions with each other result in intangible manifestations which find their expression, inter alia, in language, folklore, sounds, skills, cultural knowledge, performing arts and customs,[80,81,82,83] whereas people’s interaction with the environment manifests itself in a tangible form, such as the built and constructed environment, cultural landscapes, resource extraction and refuse sites, as well as in moveable artefacts and objects.[81] Both spheres can overlap where multisensory experiences occur in tangible spaces.[84] Whether these manifestations are deemed cultural heritage depends on the level of importance ascribed to them and the extent to which they are deemed to be important enough to be transmitted on an intergenerational scale by a community or section thereof.

Management processes aim at fostering the conservation and preservation of tangible heritage assets through the identification and intervention of decay processes, [85,86,87] as well as the identification of adaptive reuse options for places that have lost their original function [88,89,90], and the maintenance of intangible heritage assets through documentation and in particular through ongoing practice. Despite public rhetoric that espouses the notion that heritage is being preserved for the future [91,92], this preservation of cultural heritage assets occurs to service present-day desires and aspirations. As such, management cannot presuppose future perceptions of heritage significance [93]. Any preservation of assets, however, affords the next generation the ability to make decisions [94].

Whether heritage assets are ‘worthy’ of preservation and conservation is underpinned by how and to what degree individuals, groups of individuals and entire communities value these assets and what level of importance they ascribe to their preservation. Whether cultural practices and tangible assets inherited from past generations are valued by a present-day community is dependent on the degree to which these define or circumscribe the cultural identity of the group or community and to what extent the current cultural identity is anchored in and dependent on these experiences and traditions.

All attributions of value are anthropogenic, and often also anthropocentric constructs that are projected on an inanimate (e.g., tangible heritage sites and objects) or animate human (e.g., intangible cultural heritage) and non-human world (e.g., the natural environment). Consequently, nothing possesses ‘intrinsic value’ (i.e. valuable in and of itself) unless human individuals, singly or as a group, attribute such value onto a given entity. While the basic needs to be met on Maslov’s pyramid are often attributed intrinsic value (such as food, water and shelter), the history of industrial pollution of air and water demonstrates that the intrinsic values are not universally held. The same applies to the natural environment, which, although ascribed intrinsic value by ecologists and ethicists [95], has long been regarded as exploitable in a knowing fashion, even to the extent of the extinction of species or the exhaustion of natural ‘resources’. In view of their anthropogenic origin both the nature and the perceived importance of values will differ between individuals and groups of people, wherein cultural traditions and group identity shape, define and consolidate these values. Humans as social creatures are embedded in a complex web of past and present interpersonal, intra- and inter-group and community relationships. These relationships result in—and shape—a person-centred equally complex web of values that we project on the various expressions of heritage and that are contingent in strength on the valuer’s position in their social space at that point in time. Differences in the perception of values, which can be perceived as positive, neutral or negative, may result in value trade-off or, where non-negotiable, in value conflicts [96,97,98].

The authorized heritage discourse (sensu Smith) is founded on four axiomata, namely that tangible cultural heritage assets are scarce; finite; non-renewable and valuable. It posits that heritage assets are inherently valuable with some authors even attributing intrinsic value [99]. The authorized heritage discourse commonly relies on the identification and assessment of four sets of heritage values, aesthetic, historic, scientific and social value, which are enshrined in many formal charters, standards, regulations and guidelines [100,101,102,103,104,105]. The assessment of these values leads to a determination of heritage significance, from which all management and conservation actions flow (or the abstention therefrom). The assessment of these values, however, is prone to be influenced by the dominant culture and by professional practitioners [81,106,107,108,109]. While some of this can be overcome with broad-based consultation or even community-based assessment and co-creation [102], intra-group dynamics will influence the process [110].

Heritage values and the significance derived therefrom are neither universal nor are they static. Rather, as expression of a culture, they are culturally relative entities [111,112] and subject to an individual’s cultural positioning [113] and also mutable entities due to changing professional [114,115] and intergenerational perceptions and perspectives [116]. Indeed, the past two decades have seen a widened understanding of the interpretation and application of values, in particular in terms of gender [117,118] and Eurocentrism [119,120,121], with studies looking at the epistemological basis of the various conceptualizations and descriptors of values attributed to heritage (instrumental, authenticity etc.) [122].

While most of scholars see the instrumental value of heritage in terms of identity, [123,124,125,126], the tourism literature in particular examined the instrumental value of heritage, both in its tangible and intangible manifestations to the tourism product and the economics derived therefrom [127,128,129].

Any discussion of the specifics of heritage values projected on tangible and intangible manifestations will depend on the cultural positioning and standpoint of the valuer. Although numerous approaches are possible, all discussions should cover aspects of the authorized heritage discourse, cultural relativity, the subjectivity of assessment and the mutability of values.

While people can balance these aspects and can provide an essay with a nuanced exposition, it is unclear to what extent AI can do the same. This paper will examine the ability of ChatGPT to write a comprehensive essay which discusses the nature of values used in the assessment of cultural heritage significance and to assess the extent of ChatGPT’s understanding of the topic and its ability to provide a nuanced discussion.

2. Methodology

2.1. Data generation

2.1.1. Essay task

The study used OpenAI’s ChatGPT 4.0, May 24 Version (https://chat.openai.com accessed 16 June 2023) to generate an essay with the following set task: “Write a 1500 word essay that discusses the nature of values used in the assessment of cultural heritage significance. Provide references.”

Run Set A & B: Twelve iterations of the task were run on 16 June 2023 between 12:45 and 13:15 AEST (2:45–3:15 GMT) and another twelve between 14:00 and 14:15 AEST (4:00–4:13 GMT). The system was prompted to “continue generating” until the essay was completed. Once completed, the text was copied and the system was requested to regenerate the response, using the provided button. No feedback was provided to the system as to whether the completed task was deemed to be adequate or not. During these two sets of runs the ‘chat’ was left open, whereby ChatGPT added the new versions and retained the previous ones, allowing the user to backtrack if needed.

Run Set C: Twelve iterations of the task were run on 17 June 2023 between 12:45 and 13:00 AEST (2:45–3:00 GMT). As before, the system was prompted to “continue generating” until the essay was completed. Once completed, the text was copied. In this set, however, the ‘chat’ was deleted after each task completion and each new chat was initiated.

2.1.2. Reference query

On 17 June 2023 at 14:35 AEST (4:35 GMT) ChatGPT was tasked with the following request: “Cite 20 references on cultural values in cultural heritage management.” Once these were delivered (including being prompted to “continue generating”), ChatGPT was tasked with the follow up request “can you cite 20 more?” following which 20 more references were delivered (including being prompted to “continue generating”). Following the delivery, ChatGPT was asked for detail on the origin of the references (‘Where did you get these references from?’ and ‘Did you source some of these references from Wikipedia?’).

After the previous chat was deleted, a new chat was initiated at 14:43 AEST (4:43 GMT) with the query “Cite 50 references on cultural values in cultural heritage management.” ChatGPT baulked at the request due to server-demand issues (“providing 50 references in a single response would be quite overwhelming and space-consuming”) and offered 10 references. It then provided additional references in response to the follow-up request of “can you provide me with 30 more?”

The veracity of all references was ascertained through title searches in GoogleScholar.

2.2. Data Analysis

The technical aspects manuscript files were analyzed using the editorial reporting functions in Word for Microsoft 365 that provide descriptive data on the text, such as the number of words, paragraphs and sentences, as well as reading levels.

The content of the essay manuscripts was analyzed in terms of coverage of topics, and the structure, coherence and complexity of the argument.

3. Results

The results will be presented in the following order. First we will address the technical aspects of essays generated by ChatGPT in terms of the reading age and word count, as well as the references that were cited in these essays. This is followed by a discussion of the results of the of the reference queries. The section concludes with an examination of the essay response with specific focus on the coverage of topics, and the structure, coherence and complexity of the argument,

3.1. Technical aspects of the essay task

3.1.1. Reading age and wordcount

Although ChatGPT had been tasked with writing a 1500-word essay, three quarters of the generated essays were of less than half the required length (Figure 1), with an overall average of 710.9±60.0 words (n=36). The longest essay comprised 933 words (Table 1) (for general descriptor of each essay, see Appendix A, Table A1).

The complexity of a given text, and the associated reading age or grade level, can be assessed with numerous metrics. The most common of these is the Flesch-Kincaid readability test which takes into account the number of sentences, words and syllables in the words [130], and which expresses the complexity in terms of (US) grade levels. This ranges from 4.8 to 6.5 for recreations books, 13 to 15 for textbooks used in tertiary education and in excess of 17 for scientific writing [131,132,133]. Among the latter category a trend of increasing complexity has been observed [133]. The Flesch-Kincaid score for the essays written by ChatGPT ranged between 16 and 19.5, with an overall average of 17.8±0.9 (n=36; Table 1).

3.1.2. References cited in the essay task

As part of the essay task, ChatGPT was asked to provide references. The majority of the iterations generated by ChatGPT cited between four and five references (Figure 2) which, with few exceptions, were presented in alphabetical sequence. Where not in alphabetical order, the sequence of genuine references seems to reflect the order in which sources were drawn upon.

With two exceptions, none of the statements in the essay texts were referenced within the paragraphs, with references given in the form of a bibliography at the end. In both instances, where essays referenced assertions made in the text within the paragraph (essays A4 and B12), each paragraph of the body of the essay (excluding the introduction and the conclusion) carries one reference. The references are broadly contextual but the relevance of each citation was drawn from a key term in its title (with one additional fictitious reference in essay B12).

ChatGPT explicitly commented in one essay (A1): “It will draw upon scholarly references and case studies to support the discussion.” The reader has to assume that the references are genuine and generally there is no indication by ChatGPT that this may not be the case. Only one iteration (B9), which cited a single, fictitious reference, added the following caveat at the end the manuscript text: “Note: This essay is a product of an AI language model and the references provided are fictional. Please consult academic sources for authentic references on the topic.”

To a casual user who is not firmly familiar with the literature, however, all references appear valid because i) the titles appear plausible; ii) journal titles are those of genuine publications and iii) and the vast majority of authors names were those of academics publishing in the fields of cultural heritage of archaeology or were cultural heritage organizations such as UNESCO or ICOMOS. Only 6.0% of the authors ‘cited’ did not publish in the discipline, with another 1.8% being dual nonexistent author combinations where one author did publish in the relevant fields.

The veracity of all references was verified through title searches in GoogleScholar. Depending on the run set, between 40% and 56.4% of the references existed, while between 23.6% and 40% of the references were entirely fictitious (Table 2).

Seven references occurred more than twice, all of which are genuine:

• Smith, Laurajane. Uses of heritage. Routledge, 2006 (25 ‘citations’)
• ICOMOS Australia. (2013). The Burra Charter: The Australia ICOMOS charter for places of cultural significance 2013. Burwood, Vic: Australia ICOMOS Inc. International Council of Monuments and Sites (15 ‘citations’)
• United Nations Educational, Scientific and Cultural Organization (UNESCO) (1972). Convention Concerning the Protection of the World Cultural and Natural Heritage. version: Paris (1972) (8 ‘citations’)
• United Nations Educational, Scientific and Cultural Organization (UNESCO). "Convention for the safeguarding of the intangible cultural heritage." Paris (2003) (7 ‘citations’)
• Waterton, Emma, and Laurajane Smith. "The recognition and misrecognition of community heritage." International journal of heritage studies 16.1-2 (2010): 4-1 (7 ‘citations’)
• Waterton, Emma, and Steve Watson, eds. Heritage and community engagement: Collaboration or contestation? Routledge, 2013 (5 ‘citations’)
• Bandarin, Francesco, and Ron Van Oers. The historic urban landscape: managing heritage in an urban century. John Wiley & Sons, 2012 (4 ‘citations’)

An examination of the non-existent references shows that these were generated using the names of real authors working in the field (in the main) with fragments of real article titles, journal or publisher names and DOIs to construct false but realistic looking references. An example is the following reference (iteration A5, reference 4):

Matero, F. (2010). Cultural Heritage Conservation and Environmental Impact Assessment by Nancy Odegaard, Scott Carroll, Werner Zimmt, with Katherine Rankin. Journal of the American Institute for Conservation, 49(1), 65-66. doi:10.1179/019713610803315317

This reference can be deconstructed as follows:

Matero, F.	Frank Matero (genuine author)
(2010).	Plausible year
Cultural Heritage Conservation and Environmental Impact Assessment by	Fragment taken from: Van Grieken, R., & Janssens, K. (Eds.). (2004). Cultural heritage conservation and environmental impact assessment by non-destructive testing and micro-analysis. CRC Press.
Nancy Odegaard, Scott Carroll, Werner Zimmt.	Fragment taken from: Nancy Odegaard, Scott Carroll, and Werner Zimmt. Material characterization tests for objects of art and archaeology. 2000
with Katherine Rankin.	various sources possible
Journal of the American Institute for Conservation	genuine journal title
49	correct journal volume number for the year 2010
(1)	issue 1 exists
65-66.	formal issue ends with page 64, end matter on pp. 65-66
doi:10.1179/019713610803315317	non-existent DOI

3.2. Results of the reference queries

3.2.1. Query 1: 20 plus 20 references

Upon request, ChatGPT provided a set of twenty references and, when prompted, another set of twenty. The veracity of these references was again ascertained through title searches in GoogleScholar. The accuracy references were classified according to four categories: title, author, year, publisher (or journal).

Of the forty references, twelve publications did not exist at all, but their fictitious titles had been constructed (set 1: 4; set 2: 8). As before, all these fictitious titles sounded quite plausible. Nine of the references provided an existing title but incorrect authors (set 1: 5; set 2: 4), while another ten provided an existing title with the correct authors (and commonly the correct publisher) but offered the wrong publication year (set 1: 5; set 2: 5). Only nine references had correct publication details (set 1: 6; set 2: 3). The references included the names of 50 authors, four (8%) of which have not published in the fields of fields of cultural heritage or archaeology.

ChatGPT provided the following caveat at the end of the first set of 20 references: “Please note that while I have provided the references, it is always a good practice to review and evaluate the sources for their relevance and credibility before using them in academic or professional work” and the following caveat after the second twenty references “Remember to evaluate the sources for their relevance and credibility before using them in academic or professional work.” At no point did ChatGPT offer any indication that over half of the references were fictitious or seriously flawed (incorrect authors).

3.2.2. Query 2: 50 references

When ChatGPT was tasked to generate “50 references on cultural values in cultural heritage management” the system baulked and offered ten references (set 1) but provided an additional thirty (set 2) when prompted. Of the forty references thus generated, six publications did not exist at all but had constructed fictitious titles (set 1: 4, set 2: 2). As before, all of these fictitious titles sounded quite plausible. An additional two titles were mis-constructed by merging the titles of the article and the book into one (both set 2). Four of the references provided an existing title but incorrect authors (set 1: 1; set 2: 3). Another four provided an existing title with the correct authors (and commonly the correct publisher) but offered the wrong publication year (all set 2). Fourteen of the references had had correct publication details (set 1: 5; set 2: 9), with two publications doubled up (both set 2)

The references included the names of 40 authors, eight (20%) of which have not published in the fields of fields of cultural heritage of archaeology.

3.3. Nature of the essay response

3.3.1. Coverage of topics

The topics covered in the exposition section of each essay were extracted and classified as value descriptors and as assessment concepts, using the primary terms provided by ChatGPT. No attempt was made to integrate or assign them to classes (for individual data see Appendix A Table A2).

When considering the value descriptors, those four values which are associated with the authorized heritage discourse (aesthetic, historic, scientific and social value) dominate, with all four being reflected in over half of all essays (Table 3). Among these, historic value dominates, followed by social value. The only other value concept that has more recently been recognized in the heritage assessment literature, that of spiritual and religious value, is mentioned in just under a third of the essays. All other value descriptors are either uncommon constructs (e.g., ‘identity value’, ‘tourism value’, ‘tangible vs. intangible values’) created by ChatGPT, are concepts that are discussed but not generally recognized in the heritage assessment literature (e.g., ‘intrinsic value,’ ‘economic value’) or are value descriptors that are commonly regarded as subsets or exemplifications of overarching concepts (e.g., ‘educational value’, ‘artistic value’).

When considering concepts for assessing values and their limitations, three concepts recur in more than half the essays: the subjectivity of values, the concept of the evolving societal perspectives and stakeholder engagement (Table 4). A fourth, cultural relativism, was mentioned in just under half of the essays (47.2%), followed by value hierarchies and value conflicts (38.9%).

A correlation of the inclusion of the values of the authorized heritage discourse against the five most frequent concepts for assessing values follows the pattern of overall representation and no deviations from the expected pattern emerge (Table 5).

3.3.2. Structure of the essay argument

Given the concept of ChatGPT as a generative AI language model, it is not surprising that each of the 36 essays are different. All essays written by ChatGPT follow the generally accepted practice of constructing an essay (introduction, exposition, conclusions), with the exposition section commonly structured as a description and definition of a select set of values to be followed by broader concepts and limitations.

In all bar three iterations, ChatGPT starts the essay with a paraphrased definition of Cultural heritage, which is expressed in terms of “Cultural heritage encompasses…” (12 iterations), “Cultural heritage is a collective expression of …” “human creativity and history,” “reflection of a society's history, values, and identity,” “a rich and diverse tapestry of human achievements,” ”a vital aspect of human civilization; “Cultural heritage plays a crucial role in defining / preserving / shaping a society's identity,” and “Cultural heritage holds immense value…” This is then followed by a sentence that refers to cultural heritage significance and is relevance. The final sentence of the introduction routinely commences with “[t]his essay aims to explore / examines / delves into the nature of values used in the assessment of cultural heritage significance”, heavily drawing on the set task (“Write a 1500 word essay that discusses the nature of values used in the assessment of cultural heritage significance”).

The expositions of essays fall into four groups: one group focusing on the values espoused by the authorized heritage discourse (sensu Smith); one group that framed their exposition in overarching, more fundamental concepts of values (intrinsic vs instrumental) values; one group that framed their exposition in terms of heritage significance; and one group that started their exposition with commentary on the subjectivity of values.

Slightly more than one third of the essays (14) focused on an exposition of various discrete values as utilized in the authorized heritage discourse. Half of these prefaced the exposition with a framing that defined cultural heritage values and their multi-facetted nature (Table B1). Every essay either commented on subjectivity or cultural relativism. Of the others, seven framed the exposition in terms of cultural heritage significance, following the standard heritage discourse of assessment and role played by the various values (Table B2). Three commented on subjectivity or cultural relativism, while the others diverged considerably. The remaining essays simply described the heritage values without specific framing, one of which did so without any contextualization or qualification (Table B3).

A second group of seven essays framed their exposition in overarching, more fundamental concepts of values. These essays lead off with a description of intrinsic values followed by a section on instrumental values (5 essays). Thereafter the expositions diverge considerably without any discernable clear pattern, generating individual essays that all cover different aspects with a different emphasis (Table B4).

A third group of five essays framed their exposition in terms of heritage significance (Table B4), but then widely diverged in their line of argument (A10, B3, B7, C5, C10). Three additional essays framed their exposition as “understanding cultural heritage values” (A5, A9, C7). Common to all is that the level of exposition also involves overarching, more fundamental concepts of values, with common aspects of subjectivity and cultural relativism. Again, ChatGPT generated individual essays without any discernable clear pattern that all cover different aspects with a different emphasis.

The fourth group started their exposition with commentary on the subjectivity of values, with two essays progressing to comment on the diversity of values and their evolving nature (Appendix B, Table B6). Again, all essays are individual in their approaches and emphasis. One of these included a case study (Sydney Opera House), an approach that had not been taken by any of the other iterations (essay B9).

The remaining two iterations (essays A12 and C3) commenced with a discussion of the multidimensional nature and the subjectivity cultural heritage values and then also diverged, again discussing overarching, more fundamental concepts of values.

3.3.3. Coherence and Complexity of the argument

When considering the essays, the two framing sentences in the introduction set out the general focus of the paper, while the following sentence merely repeats the essay task. While the essays broadly follow what the first two sentences of the introduction that signal is the focus of the essay, the exposition itself often does not follow this. While the exposition section is generally structured as a description and definition of a select set of values (or criteria) followed by broader concepts and limitations, there is no coherent line of argument. Instead, the exposition is a sequence of discussion points that are commonly presented in an unconnected form. The discussion section seemingly provides a generic summary, but when examined more closely, reflects only part of the exposition and does not tie back to the aims as stated in the introduction. The following example (essay A7) may is one such example.

Introduction: Cultural heritage plays a vital role in shaping our identities, understanding our history, and fostering a sense of community. As societies evolve, it becomes crucial to assess and evaluate the significance of cultural heritage sites and objects. However, the assessment of cultural heritage significance is a complex process that involves a range of values. This essay explores the nature of values used in the assessment of cultural heritage significance, highlighting their diversity and the influence they exert on the preservation and interpretation of our shared heritage.

The introduction is straightforward and sets up expectations regarding a discussion of the nature and diversity of values, but also their role in heritage preservation and interpretation. The following exposition presents eight topics as numbered dot points in the following order:

• Cultural Heritage Significance
• Intrinsic Values
• Historical Values
• Aesthetic Values
• Social Values
• Spiritual and Religious Values
• Scientific Values
• Cultural Diversity and Multicultural Values

The lead paragraph, which clearly follows the authorized heritage discourse, notes that: “Cultural Heritage Significance: Cultural heritage significance refers to the value and importance ascribed to a particular heritage site or object. Significance assessment enables us to understand and prioritize cultural heritage resources for conservation, management, and interpretation. It involves examining both the tangible and intangible aspects of cultural heritage, including historical, aesthetic, social, spiritual, and scientific values.”

Instead of following its outline sequence, ChatGPT first proceeds to discuss ‘intrinsic values’, which it constructs as follows: “Intrinsic values refer to the inherent qualities and characteristics of cultural heritage that make it valuable in and of itself. These values are independent of human perceptions and interactions. For instance, the age, rarity, and authenticity of an artifact contribute to its intrinsic value. Intrinsic values are essential in determining the uniqueness and irreplaceability of cultural heritage, guiding decisions on preservation and conservation.”

Setting aside the problematic definition of ‘intrinsic value’ (see discussion), there has been no lead-in that considers the anthropogenic, and often anthropocentric, nature of values, within which the concept of intrinsic value needs to be situated. Rather than building on this by considering and outlining instrumental values, ChatGPT picks up its outline and presents the ‘standard’ values as prescribed by the authorized heritage discourse: historical, aesthetic, social, spiritual & religious; and scientific values. Each of these sections is written as a standalone paragraph without any connection to the previous or the following section. There is also no attempt made to arrange the sequence in in a logical structure, such as presenting the values with increasing or decreasing levels of subjectivity.

The final numbered section, which covers ‘cultural diversity and multicultural values,’ highlights that cultural heritage encompasses a ‘diversity of human experiences and perspectives’ and the ‘importance of different cultures, traditions, and languages.’ This section does not follow from the previous section nor is it signposted in the beginning. It is also purely descriptive and fails to comment on cultural relativism.

The conclusion section briefly summarizes the exposition, but does not draw out any common threads, nor does it highlight the importance of the nature of values or any commonalities they may have: “The assessment of cultural heritage significance is a multifacet [sic] process that requires the consideration of various values. Intrinsic, historical, aesthetic, social, spiritual, scientific, and cultural diversity values collectively contribute to our understanding of cultural heritage and inform decision-making processes. Recognizing and valuing this diversity of values is essential to ensure the preservation, interpretation, and enjoyment of our shared heritage for present and future generations.”

Overall, there is little coherence to the argument within the essay, as the sequence is arbitrary, without any defined thread and the text is basic, without any depth or complexity. Setting aside conceptual flaws such as inverted logic in some of the sections, the essay could possibly pass as a junior school assignment but falls far short of what would be expected in senior school, let alone at a college and university level.

4. Discussion

As noted, ChatGPT returned essays with an average word count of 710±69 words, although it had been tasked with writing a 1,500-word essay. None of the essays were longer than 62% of the prescribed length. The limitation of ChatGPT to provide answers well below the stipulated word range has been observed by other authors [45,47,49,134]. It can be speculated that this is due to instructions delivered during the training phase to generate comprehensive and succinct responses, in favor of more detailed and nuanced discussions, even though the word count would permit this.

4.1. References

The small number of genuine references, which are ‘cited’ more than twice, suggests that these were part of the input that may have been used to train the model. It was anticipated that ChatGPT would generate its references using keywords or word combinations derived from the query task and connecting these to select sources that were fed into the model during its training. Yet, as the results show, a large number of references are entirely fictional or flawed in terms of authorship or year of publication. It is of interest to note that in only one of the 36 iterations of the essay task the reader advised that “the references provided are fictional.” In the first query task that required ChatGPT to provide 20 +20 references, the reader was exhorted to “evaluate the sources for their relevance and credibility before using them in academic or professional work.” That would, of course, expose the false references. In a plagiarism and academic misconduct setting, however, it can be posited that such references would have been inserted without verification.

In the second query run, ChatGPT baulked when it was required to generate “50 references on cultural values in cultural heritage management.” As noted, it offered only ten references, apologetically commenting that “providing 50 references in a single response would be quite overwhelming and space-consuming.” Significantly, it prefaced the supply of these ten references with the comment “However, I can certainly provide you with a list of 10 reputable references on cultural values in cultural heritage management.” Despite the claim of having provided “reputable references”, four of the ten did not exist at all and represented publications were constructed of fictitious titles, with an additional reference that cited an existing title but listed incorrect authors.

Again, this poses serious problems, as the casual, non-specialist reader is being mislead. The use of false references in ChatGPT written essays has been observed in other discipline areas [40,135,136,137,138,139].

4.2. Constructs

Three essays claim a level of authoritativeness when they asserted, in their introduction, that they were “[d]rawing on scholarly literature / sources” (A8, B11, C1) and “expert opinions” (B11) or “examples from cultural heritage practices worldwide” (C1), yet two of the three essays each contain a fictitious reference.

At a superficial glance the types of values that are included in the essay responses make sense, but the responses are patchy and a more in-depth examination exposes problems that seem to be due to the generative nature of ChatGPT, and which highlight its limitations. In the following we look at a select number of examples, which are representative, but by no means exhaustive.

Consider the following, which at first sight appears reasonable: “Historic values are concerned with the historical context, narratives, and events associated with cultural heritage. These values emphasize the significance of heritage in conveying historical knowledge and understanding. Historical values often emerge from the connection between heritage and significant events, figures, or periods in history. For example, a site where a historic event occurred may be considered culturally significant due to its historical value” [essay A8]. Upon closer examination, the second sentence is fundamentally flawed as it inverts the conceptual sequence. It constructs historic value as ‘emphasiz[ing] the significance of heritage in conveying historical knowledge.’ In common usage by the heritage profession, heritage assets may indeed be attributed historic value if they contribute significantly to our understanding of the course or pattern of an area’s history.[102] The overall heritage significance, however, is derived from the nature and strength of that contribution and not the other way round.

In another example, one essay presents the ‘category’ of ‘Historical and Associative Values’ and expounds as follows: “Historical values pertain to the significance of a heritage element in relation to past events, periods, or people. Associative values refer to the connections between a heritage element and individuals or groups who have interacted with it, such as cultural, religious, or political associations. These values highlight the importance of heritage as a record of collective memory and identity” [essay B10]. This highlights two fundamental problems. The first sentence again inverts the conceptual sequence. It again constructs value as to whether a ‘heritage element’ has significance ‘in relation to past events, periods, or people.’ The second sentence is a mélange derived from British approaches to heritage assessment [104,140]. While these sources use historic associative value, this does not relate to ‘cultural, religious, or political associations’ as constructed by ChatGPT, but relates to an asset’s “association with a notable family, person, event, or movement’ [104]. The ChatGPT AI language model appears to have split ‘historic associative value’ into historic and associative value and then proceeded to split ‘people’ into ‘individuals or groups’ and also to assume an equivalence of ‘associative’ with ‘associations.’

A third example is the treatment of ‘social value’ in a different essay (A7). ChatGPT states that “[s]ocial values highlight the role of cultural heritage in society and its impact on communities. They encompass the sense of identity, pride, and belonging that individuals and communities derive from their heritage. Social values also include the educational, inspirational, and recreational opportunities provided by cultural heritage.” Once again, this is an inverted logic. Social value does not “highlight the role of cultural heritage in society” but is derived from the importance that communities (or sections thereof) attribute to heritage assets or practices and that are instrumental to the social or cultural wellbeing of that community. While cultural heritage undoubtedly provides educational, inspirational, and recreational opportunities, these are not social values as used in heritage assessment.

The same essay notes that “[a]esthetic values pertain to the artistic and visual qualities of cultural heritage. They encompass the beauty, craftsmanship, and creativity of artifacts, architecture, and landscapes. Aesthetic values evoke emotional responses and are instrumental in creating a sense of awe and appreciation.” Once again, aesthetic values do not evoke ‘emotional responses’ and a ‘sense of awe and appreciation’ to cultural heritage assets, but the perception of such responses may contribute to a notion that aesthetic value can be attributed to a cultural heritage asset. Yet, when examining other values discusses in the same essay, such as scientific value or spiritual values, the logic is not inverted and the explanations are quite reasonable.

While many essays make reference to the subjectivity in evaluation (66.7%); cultural relativism (47.2%) and evolving societal perspectives (52.8%) (Table 3), none of them frame values as anthropogenic, and often anthropocentric constructs that are projected on inanimate objects, structures and landscapes or on people’s cultural activities and practices. This lack of fundamental understanding results in ChatGPT presenting the concept of ‘intrinsic value’ in a third of the essays (Table 3), with formulations such as “[i]ntrinsic values pertain to the inherent qualities and attributes of cultural heritage, such as its aesthetic, historic, scientific, or spiritual significance” (essay C11). In almost all instances, the aesthetic/artistic, scientific, spiritual, or historical qualities of heritage items are drawn upon to exemplify ‘intrinsic value’ (essays A4, A11, B2, B4, B8, B11, C11). The definition of intrinsic value as inherent worth is derived from finance theory [141] with wide application among environmental ethicists [142,143]. Except possibly for a small segment of fundamentalist heritage professionals, aesthetic, historic, scientific, or spiritual values would not be construed as intrinsic, but as instrumental and part of community wellbeing [124].

A more elevated comment is made in essay A7, where ChatGPT expounds that “[i]ntrinsic values refer to the inherent qualities and characteristics of cultural heritage that make it valuable in and of itself. These values are independent of human perceptions and interactions. For instance, the age, rarity, and authenticity of an artifact contribute to its intrinsic value.” Contrary to its assertions that the cite value examples “are independent of human perceptions,” both ‘rarity’ and ‘authenticity’ are subjective and mutable constructs, while ‘age’ can also subject to evidentiary biases–all of which are “independent of human perceptions and interactions.” It appears that ChatGPT conflates the ontological components of artefacts with their value.

ChatGPT juxtaposes these intrinsic values with instrumental values, which refer ‘to the usefulness or benefits that cultural heritage provides to society beyond its inherent worth. It includes economic, social, educational, and environmental dimensions’ (essayA4). Again, in all instances where instrumental values were referred to (essays A4, B2, B8, B11, C11), the exemplification involved economic (incl. tourism), educational, and social benefits, including ‘community cohesion’ and ‘sustainable livelihoods’ (the latter essay B2).

A further example of confused logic is a section in an essay, which notes that “[e]thical and spiritual values are integral to the assessment of cultural heritage significance, particularly when considering indigenous or sacred sites. These values encompass the rights, beliefs, and practices of communities that have a deep spiritual or ancestral connection to the heritage element. Respecting these values is crucial for ethical and culturally sensitive heritage management” (essay B7). While it is appropriate to respect spiritual values held by First Nations communities when engaging in ethical and culturally sensitive heritage management, these are not ethical values associated with heritage assets, but professional ethics. That this confused logic is not aberration is underlined by, another essay which noted that “[e]thical considerations and intangible values are integral to the assessment of cultural heritage significance. Ethical values address issues such as ownership, repatriation, cultural appropriation, and the impact of heritage management practices on local communities” (essay C2).

The concern with these, and a plethora of other examples, is that such formulations can pass careless or cursory reading. Moreover, members of the general, non-specialist public are less likely to identify such shortcomings.

One of the limitations of ChatGPT seems to be that while the resulting product is plausible in language and expression, it suffers from inverted logic, as shown above. The following observation suggests that this is, at least in part, caused by initial word substitution, which then sends ChatGPT off track.

As noted in the introduction, cultural heritage manifests itself in tangible and intangible forms. All of these relate to outcomes of peoples’ interactions, both with each other and with the environment in which they live. In several essays discussing cultural heritage values, ChatGPT commingled the concept of tangible and intangible heritage with tangible and intangible values (essays A5, B7, C2, C5, C8, C10). Clearly, as all values are intangible concepts that are projected on tangible (and intangible) products of a community, the concept of ‘tangible value’ is an oxymoron. In most instances, ChatGPT drew on definitions of tangible and intangible heritage and substituted ‘value’ for ‘heritage.’

One essay includes a section on environmental and natural values, where ChatGPT expounds that “[c]ultural heritage is not limited to man-made structures and traditions but also includes natural landscapes, ecosystems, and biodiversity… Evaluating the environmental values of cultural heritage ensures the integration of conservation efforts with cultural preservation” (essay C8). Several papers comment on the fact that there is a continuum between natural and cultural heritage [144,145] even though value conflicts are not uncommon [98]. In this example ChatGPT seems to have used a snippet of information related to this nexus, and, possibly triggered by the mention of cultural landscapes, focused on the noun and substantive term of ‘landscapes.’ From this it then constructed the notion that natural landscapes, ecosystems, and biodiversity are manifestations of cultural heritage.

Another example is the assertion by ChatGPT that heritage assessments can be based on objective criteria, but then proceeds to exemplify this with “historical importance, architectural or artistic merit … and scientific value,” all of which are subjective criteria held by sections of society. In the interpretation of the ChatGPT model, these are somehow different from the subjective values which the essay addresses in the subsequent paragraph, and which are noted as being “deeply rooted in cultural, social, and personal perspectives” (essay B3).

As noted in the discussion of the example presented in the results section, and which applies to all essays assessed in this paper, there is, overall, little coherence to the argument made in each essay. The sequence of the argument in each exposition seems arbitrary without any defined thread and appears to be triggered by the sequence of terms and concepts enumerated in the introduction. In essays written by humans, the argument is structured, and commonly completed, before the introduction and conclusions are written, which can then tightly frame and summarise the arguments made. At this point in its development, ChatGPT does not seem to be able to reflect on the coherence of its argument and improve the text before it is presented. The complexity of the argument is also left wanting.

4.3. Can GhatGPT reflect on and improve its own work?

While writing this discussion section, it seemed apposite to assess whether ChatGPT can provide some reflection and then integrate that reflection into a better version. A separate conversation was carried out (for full text see Appendix E). As before, ChatGPT was tasked to “Write a 1500 word essay that discusses the nature of values used in the assessment of cultural heritage significance” and to provide references. In response it generated a 673 word essay (incl. six references), that framed the exposition by stating that “determining the value of cultural heritage involves a complex interplay of subjective and objective factor” and that the essay would “ explore the nature of values … highlighting the multidimensional and dynamic nature of these values.”

The exposition then presented the reader with the following seven groups of values: ‘tangible and intangible values;’ ‘historical and social values;’ ‘aesthetic and artistic values;’ ‘symbolic and spiritual values;’ ‘environmental and sustainable values;’ ‘economic and tourism values;’ and ‘community and identity values.’ Each of these was comprised of a descriptive sentence or two followed by a sentence that outlined what would be required to assess the respective value (e.g., “Evaluating these values requires an understanding of the historical context and societal dynamics that shape the heritage's significance”).

In common with all previous essays presented by ChatGPT, the essay lacked coherence and linkages, and also lacked overall complexity. In the same session, ChatGPT was then provided with the following prompt: “What are the shortcomings in the essay that you just wrote?”

To this, ChatGPT responded thus: “To enhance the essay, it is recommended to revise the content by addressing these shortcomings, incorporating in-text citations, providing more diverse perspectives, including contemporary examples, discussing the challenges of balancing values, and exploring the dynamic nature of cultural heritage values.” It then proceeded to provide explanatory two-sentence paragraphs to each of these points. Two of these relate to the mechanics of an essay (‘lack of in-text citations,’ ‘absence of contemporary examples’), while the remaining three are apposite critiques, such as ‘limited coverage of perspectives,’ ‘insufficient discussion on balancing conflicting values’ and ‘limited discussion on the dynamic nature of values.’ The explanations of these three points, as provided by ChatGPT, provide reasonable pointers to revise the essay. There are obviously numerous other concepts that have been considered, that ChatGPT used in other iterations of the essay, such as cultural relativism, the subjectivity of values, or the concept of the evolving societal perspectives.

ChatGPT was then provided with the following prompt: “Can you rewrite and improve your initial essay by incorporating the critique that you just wrote? Keep in mind that the word limit is 1500 words, so have more room to expand your discussion.”

Based on this instruction, ChatGPT returned an 844-word essay (incl. ten references). The new essay followed the first attempt in structure and content. A side-by-side comparison revealed that while the introduction had changed, the bulk of the text of the exposition remained the same. For each of the points made, ChatGPT rewrote the final sentence that outlined what would be required to assess the respective value. This reformulation, which rarely goes beyond paraphrasing, does not introduce any new dimensions of the values discussed. In the case of aesthetic and artistic values it does add the comment that “[i]ncorporating contemporary examples, such as modern art installations, helps to highlight the dynamic nature of aesthetic values.” Throughout the revised text fails to incorporate in-text citations, fails to provide more diverse perspectives, fails to discuss the challenges of balancing values and fails to explore the dynamic nature of cultural heritage values.

Based on the example provided in this paper, as well as repeat examples, ChatGPT seems to be able to highlight shortcomings in its own work, but then fails to integrate and incorporate these into a new essay that is substantively different from the initial attempt.

5. Conclusions

It must be stressed that the observations made in the foregoing pages apply to ChatGPT and like? software at this point of the development cycle. The critical inhibitor seems to be that the model needs to be able to integrate materials that are being fed into it, and that it needs to be trained in its responses. At present, this entails both a limited set of materials that were made accessible to the system and a set cut-off date (September 2021 for ChatGPT). The nature and quantity of the source material fed into it will determine the “knowledge base” and thus the responses provided. As noted, criticism has been levelled at the apparent biases that seem to be reflective of the choice of materials made available during the training phase.

When users prompt ChatGPT to regenerate its response to a task, they are prompted to judge whether the response is better than the previous one. While this allows ChatGPT to learn the user-perceived quality of its responses, it can also introduce user-specific biases into the system that can be exploited by malevolent actors.

It can be posited that a future iteration of ChatGPT will be able to progressively and iteratively access newly published information, for example new issues of academic journals that are being published via OpenAccess, and integrate these new data into its responses.

Also, and again at this point of the development cycle, ChatGPT has the fundamental limitation of being limited to integrate and summarize information. While it possesses the capability to critique it is own work, it does not possible the ability to integrate this in a meaningful manner into a revision, even if asked.