This is actual a pretty old and often debated question. It is called "Lady Lovelace's Objection" and first appeared in Alan Turing's seminal paper "Computing Machinery and Intelligence".
Below is my response to Lovelace's Objection, as well as Alan Turing's response which I wrote for a philosophy course in 2015. Perhaps it will be of interest to you?
Turing created the imitation game as a way to talk about cognition in
machines without being bogged down by the philosophical history of the
word “think” and “machine”. In his discussion of the imitation game,
he addresses Ada Lovelace's objection and refutes one aspect of it,
however he does not cover all aspects. Namely, his rebuttal to
Lovelace's objection seems to assume the universe is deterministic and
by rephrasing Lovelace's argument, he may be missing some of it's more
subtle implications.
Turing's imitation game has the objective of causing a human judge to
be unable to correctly guess which amongst two players is a computer.
The only tool at the judge's disposal is textual communication that
can be directed to each of the players individually, similar to an
Internet chat room or an e-mail exchange. If the machine can confuse
the judge, given a certain time or question limit, the machine is said
to have won the imitation game and to have passed the Turing test.
Lovelace's argument against a machine passing the Turing test and thus
being able to “think”, is that a machine (specifically referring to
the Analytical Machine) “has no pretences to originate anything. It
can do whatever we know how to order it to perform” (Turing 1950,
450). Turing faces this argument by first rephrasing it into
questioning whether a machine “can never do anything 'new'” (Turing
1950, 450). He refutes this reformulated objection by questioning
whether human beings have ever done anything new. In other words, he
claims that there is no verifiable proof that any creation from a
human being was not the product of their education; whether formally
studied or learned from the environment.
Turing's claim is a reasonable one, but he risks getting caught up in
the debate of free will versus determinism, since he seems to be
claiming that a human being is purely a function of this environment.
He assumes that all that is necessary to imitate a human being is to
determine the function that maps the external environment to human
reasoning. The belief of the existence of this function is known as
determinism within philosophy and is a matter of much debate.
Alternatively, to avoid this philosophical quagmire, instead of
appealing to determinism, we could instead look at what is required
for a computer to create something “new”. Creating something “new”
could be interpreted as synthesizing new information. The proof that a
computer can accomplish this task can be proven by examining a
rule-based system.
A rule-based system is based on the idea that most of human knowledge
can be represented by rules and human reasoning can be approximated by
the manipulation of these rules in a logical manner. For example, to
represent a navigation plan, you can know a series of logical steps or
rules describing routes, such as “Lester street connects my house to
highway 69” and “highway 69 leads to Sudbury”. It is possible to
program a computer using to search through these steps and come to a
conclusion, in this case the route from Sudbury to Lester street
(Thagard 2005, 47). You can then synthesize new rules by combining
them using a process known as “chunking” or “composition”, and saving
them so that the search doesn't need to be executed every time this
objective (getting from Sudbury to Lester) needs to be accomplished
(Thagard 2005, 49). Additionally, rules can be used to generate
explanations or hypotheses by abductive inference (Thagard 2005, 50).
Both the plan, the explanation and the summary did not exist until the
computer sought them out and discovered them. This can be seen as the
equivalent of a machine producing something “new” and acquiring “new”
knowledge. Using mathematical terminology, it can be said that the
system is non-monotonic.
There are many flaws to this simplistic rule-based system. It is
computationally inefficient and does grasp all of the psychological
complexity of a human being. Consequently, it could be argued that
this model could never evolve to successfully imitate a human being.
However these are superficial concerns. It would be a mistake to
broaden this specific rule-based system's inability to evolve to all
computational models. Instead, what should be retained from the
example of the rule-based system is that it is possible to contain
aspects of human reasoning and ingenuity within a computational model.
Additionally, the flaws of computational inefficiency and
psychological incompleteness can be and have been addressed by newer,
more complete models. For example, ACT-R has been shown to be able to
imitate certain attributes of a human being, such as natural language
processing, in the form of language acquisition (Anderson 2002, 1).
References:
- Alan Turing (1950), Computing Machinery and Intelligence
- Paul Thaghard (2005), Mind: Introduction to Cognitive Science, 2nd Edition. MIT Press.
- Taatgen, N.A. & Anderson, J.R. (2002). Why do children learn to say "broke"? A model of learning the past tense without feedback. Cognition, 86(2), 123–155.